Hardware requirements for 10G routing?
-
This topic comes up from time to time, and I see just yesterday someone's asked about using pfSense as a cheap 10G router in a specific situation.
My question is similar but from a slightly different angle. I already have a bunch of Chelsio T420-CR cards. I'm assuming that there is some level of pfSense hardware platform spec that will handle routing at these speeds, although it'll never be as fast as an ASIC based platform of course.
Unfortunately the website's hardware pages don't give any guidance about what kind of platform spec might be suitable for various workloads of 10G-containing networks, compared to the guidance provided for <=1G networks, but I would imagine quite a few people in the core team, and probably others, have experience with using pfSense on 10G to beat the 1GBE bottleneck (as opposed to using multipathing/aggregation). Basically if I'm willing to throw modern higher-spec hardware at pfSense for a small network with data rates between cards peaking up to say 4GB/s and jumbo frames used, what level of spec should I consider appropriate, and what's your experience of how it will work out?
For example, do AES extensions help? Is the limiting factor the single core speed or the multithreaded total across all cores? Which generation/families of CPU should be considered if 10G routing will be a significant network task? Which generations of chipset contains significant features for 10G scenarios (bus speeds or other technologies)? If the main traffic goes between two NICs does it make a difference if the two NICs are physically on different PCIe cards, on the main board, or on the same dual-port PCIe card? How badly would latency work out for pure switching tasks compared to a dedicated switch and to what extent can the platform minimise this?
There are probably users and situations for which it makes more sense to run pfSense on a higher-spec platform with 10G cards, than buying separate dedicated routing/switching hardware.
So I'm hoping to stimulate a good discussion of suitable hardware for 10G routing, as well as a specific indication of hardware spec for myself. I appreciate that it will raise questions about the need for 10G and the benefits of dedicated hardware, but as the pfSense store itself lists 10G pfSense hardware, my hope is that I'll get good quality information rather than being distracted :)
-
Sounds like the platform you're describing is xeon -D + 10gbe but this platform is usually geared towards virtualization labs aka hypervisor. Dose this pfsense box only need to exist as the physical switch on the 10gb or is there more to it. Please provide more detail about you're network what's the workload on the 10gbe how many clients exist on this network webservers ovpn etc.. A network diagram would help
-
afaik, there isn't a system on the planet, that runs pfSense, that will do 10Gbe wirespeed(=small packets). it can't be done while using the traditional kernel.
https://blog.pfsense.org/?p=1866
Using a Xeon E3-1275 (4 cores @ 3.5GHz) FreeBSD -CURRENT can forward at a rate of around 1.058 Mpps. Turning on fastforwarding (or building a kernel with tryforward support) increases this rate to about 1.33Mpps. While this is enough to ‘fill’ a 10Gbps link with full-sized frames, not all frames are full-sized, and the true test of a router is it’s ability to forward a mix of traffic, throttled only by the speed of its network interfaces.
more recent:
https://forum.pfsense.org/index.php?topic=113862.msg634832#msg634832 -
If someone will be on the safe side, to gets success and clean 10 GBit/s routed without any trouble she/he should be using a Layer3
Switch and if switching only is needed a Layer2+ switch would be the top of that roof and fairly the best bet at this time as I see it
right now. There are some nice and cheaper models from Netgear that are nice playing together in networking.
Netgear XS708T, Netgear XS712T, Netgear XS716T, Netgear XS724T, Netgear XS748T
Netgear M4300 series