Recommended Hardware for routing 300+ customers?



  • I would first like to start this post with an apology for my ignorance. There's a lot I have to learn about PF Sense and routing.

    I recently invested in a WISP and am trying to track down a few network issues, as well as upgrade some antiquated equipment.

    Our network is privately routed and made up of 4 towers that all backhaul to one central tower which has our router and DNS servers. All five towers are on separate VLANs.

    Our router is an older dual core intel with 2 gigs of DDR2 running PFSense 2.2.4. All quite old, however until recently it got the job done very well.

    A few weeks ago the A/C unit in the shack at the base of the tower iced over and air temps soared as high as 115 degrees for much longer than they should have. Ever since then we've had issues with latency on any data passing through the Core Router.

    Regardless of whether this is the cause of the latency, I feel a hardware upgrade for the core router is in order. We typically have between 350~400 customers and most nights about 70% of them are online for 2~3 hours.

    I'm just hoping someone may be able to help steer in the right direction, any help or input is much appreciated.

    Sidenote: I did do some basic searching but wasn't yielding a whole lot of results because I wasn't really sure what to look for.



  • If the old intel core2duo already did a good job, virtually any hardware on market can do a good job, ALIX APU2C4 should be fine.



  • The number of customers doesn't matter, what they're doing is what matters. 300 customers might be capable of 300 Gbps or 3 Mbps (and even bandwidth isn't really pertinent as it's all about pps, but can guesstimate pps from bps).

    I wouldn't run an APU of any sort as a core router, though hardware along those lines would be fine for remote towers.

    How many NICs do you need? What type of throughput is typical?

    Changing the hardware is unlikely to help. Any heat damage would be extremely unlikely to result in increased latency. It would most likely result in stability problems where the system ended up spontaneously rebooting, or completely stops working. Sounds like the existing system is a bit long in the tooth, so it definitely isn't a bad idea, but I'd also recommend pursuing the source of the latency separately.



  • During peak hours we typically are routing 100~150 Mb/sec. PPS is usually around 10k. As of now all the remote towers come into a VLAN switch to get tagged and then they go into one NIC on the core router. We have considered re doing this so each tower had it's own NIC on the core router.

    PFSense has be doling out crash reports pretty regularly since our heat experience. We've had other intermittent buggy issues with the interface and data recording.

    I didn't figure the latency was necessarily directly linked to our overheat, but the thing needs upgraded so I figured it was a good place to start.



  • Frequent crash reports, assuming they're kernel panics, would definitely be one of the signs of hardware failure post-overheating.

    For hardware choice, depends on how much scalability you want to build in. With only 10Kpps, 100-150 Mbps, a SG-2440 would give you plenty of overhead to scale to roughly a gigabit (at the average packet size of typical WISP environments).
    https://store.pfsense.org/SG-2440/



  • buy a prebuilt system from the pfsense store, or perhaps build a DIY supermicro atom based rig with ecc ram.



  • Thanks for the pointers guys! We're currently looking at the pair of SG-8860's. We think they'll have great long term scalability, and we'll be able to put each tower on it's own interface which is something we've been discussing for quite some time.

    Thanks!



  • We're currently looking at the pair of SG-8860's.

    Might be also a really good choice for your needs and is able let you grow up.



  • @Harkness212:

    Thanks for the pointers guys! We're currently looking at the pair of SG-8860's. We think they'll have great long term scalability, and we'll be able to put each tower on it's own interface which is something we've been discussing for quite some time.

    Thanks!

    Yep.  Definitely do a pair, and definitely something with fans.  A proper 1U server with fans can survive high ambient temps much longer than passively cooled or even regular off the shelf hardware with fans.  Of course, the fans will be screaming, but that's what they're for. I'd say go as far as get something with IPMI so you can monitor the hardware remotely, and the IT Watchdog sensor devices are awesome at room monitoring and relatively cheap.



  • @Harkness212:

    A few weeks ago the A/C unit in the shack at the base of the tower iced over and air temps soared as high as 115 degrees for much longer than they should have. Ever since then we've had issues with latency on any data passing through the Core Router.

    Regardless of whether this is the cause of the latency

    This is outside of the pfsense scope, but look at your switch or switches as well. They may very well have begun to experience issues due to excessive heat.

    And of course, your radio gear, but you'll have to go to the vendor for that kind of knowledge.

    If you want help with a failover scenario with redundant pfsense boxes and redundant switches, I'd be glad to help.



  • @BlueKobold:

    We're currently looking at the pair of SG-8860's.

    Might be also a really good choice for your needs and is able let you grow up.

    Yeah, never hurts to add speed, RAM and interfaces.



  • We typically have temperatures in the shack around 70 degrees despite the Core router, the backup router, our DNS server, and our DHCP servers all running in there. As well as a 24 Port Netgear switch, and two 8 Port switches.

    The AC Unit got low freon and only the lower bit of the condenser was being used. That in combination of very high humidity cause the already inefficient unit to ice over and quit blowing cold air. We replaced the 8~15 year old A/C with a new one and temps in the shack have been a lot more stable. We do have a temperature monitor in the shack that's supposed to send email notifications whenever temperatures exceed 75 degrees.. however for some reason it failed to send an email. That's still being looked into.

    We really like the idea of the dual racks because they seem like they'd have more than enough power to run DHCP, DNS, and handle our routing. As well as eliminate all of switches nearly since we would be consolidating 4 or so devices into 1.

    We do believe the latency is coming from the chain of netgear switches, or at least may be part of the issue.



  • @Harkness212:

    I would first like to start this post with an apology for my ignorance. There's a lot I have to learn about PF Sense and routing.

    I recently invested in a WISP and am trying to track down a few network issues, as well as upgrade some antiquated equipment.

    Our network is privately routed and made up of 4 towers that all backhaul to one central tower which has our router and DNS servers. All five towers are on separate VLANs.

    Our router is an older dual core intel with 2 gigs of DDR2 running PFSense 2.2.4. All quite old, however until recently it got the job done very well.

    A few weeks ago the A/C unit in the shack at the base of the tower iced over and air temps soared as high as 115 degrees for much longer than they should have. Ever since then we've had issues with latency on any data passing through the Core Router.

    Regardless of whether this is the cause of the latency, I feel a hardware upgrade for the core router is in order. We typically have between 350~400 customers and most nights about 70% of them are online for 2~3 hours.

    I'm just hoping someone may be able to help steer in the right direction, any help or input is much appreciated.

    Sidenote: I did do some basic searching but wasn't yielding a whole lot of results because I wasn't really sure what to look for.

    We almost have the same numbers of users. In my case its an average of 250 users to 300 peak users. Previously we had our pfsense on a core i3 2100 with 2 gigs of ram. motherboard is intel made with a single INTEL gigabit nic using a HP procurve switch to multiply the ports using VLAN.. As far as I can say that was the most stable & reliable setup we made. Then I transfer pfsense to a esxi then eventually we need our esxi rebooted without affecting the router so I have to move it again but this time to a lowly sempron 140 with nvidia lan card that crashes every now and then for unknown reason.

    In short if you ask me pfsense looks stable on intel chipsets and anything above Intel Sandy Bridge CPUs. best to take the low power Celeron or Pentium based on sandy bridge and newer with AES instructions support.



  • 115 F or C?

    I doubt anything got messed up.

    Google themselves run at 95F.

    Maybe you're chasing a ghost?



  • @W4RH34D:

    115 F or C?

    I doubt anything got messed up.

    Google themselves run at 95F.

    Maybe you're chasing a ghost?

    115F I'm sure, 115C and it would have literally melted the box. :) That's definitely hot enough to cause damage, especially on a system that's probably about a decade old, hence its fans probably not moving as much air as they used to. The max operating temp spec on most dual core desktops is around 95F.

    Google's most efficient datacenters run at a max operating temp of 95F. They're not running a decade old desktop-class hardware either, and do careful design to make that work.



  • @cmb:

    @W4RH34D:

    115 F or C?

    I doubt anything got messed up.

    Google themselves run at 95F.

    Maybe you're chasing a ghost?

    115F I'm sure, 115C and it would have literally melted the box. :) That's definitely hot enough to cause damage, especially on a system that's probably about a decade old, hence its fans probably not moving as much air as they used to. The max operating temp spec on most dual core desktops is around 95F.

    Google's most efficient datacenters run at a max operating temp of 95F. They're not running a decade old desktop-class hardware either, and do careful design to make that work.

    True enough.

    I'm not there so who knows.  I'd be curious to see what the on die temps are compared to that ambient temp.
    However there may had been a dust situation I'm not aware of.  There's a ton of factors to consider.

    If I don't smell ozone then I assume it's fine until of course something tells me otherwise.
    Most of the devices go through a flow oven over 200 degrees when they're made.

    Good time to ask how to do a filesystem integrity check.



  • Yea, it was 115 F. And it wasn't momentarily either, it was for a few hours at the least.


Log in to reply