Minimum hardware to do symmetric gigabit wan + pass 802.1x traffic to AT&T?
-
I just moved into a new place that has AT&T "GigaFiber", which is symmetric 1GBPS.
I've come to learn that I can bypass AT&T's crippled "router-gateway" device by plugging a pfSense box straight into the ONT, and letting the router-gateway still talk to AT&T to authenticate and let me on the network (as described here : https://strscrm.io/bypassing-gigapowers-provided-modem.html). I tried doing this with a EdgeRouter Lite, but I can't get anywhere near gigabit speeds with it (because bridging isn't hardware accelerated), so I'm looking at building a pfSense box to serve this role.
I want to make sure that whatever I build is capable of passing full gigabit, but don't need any special features like snort, VPN, etc… Just basic NAT, firewall, UPNP.
I'm guessing that a low-cost device like this one (https://www.amazon.com/gp/product/B01AJEJG1A) won't be capable of these speeds, so I had considered building a box with an i5-7600T.
Will a quad-core chip like the i5-7600T be enough to attain the speeds I'm looking for (while also passing that authentication traffic back to the ISP)? Is it overkill? What would you all recommend?
This is the build I'm considering: https://pcpartpicker.com/list/G98Gpb
If that's overkill, what about using something like ESXi to run pfSense (and other services) in a virtual machine? I know it's possible, but is it a good idea (in the eyes of the pfSense community)?
-
bump for this question.
2 ericseastrand: I can tell you though that I am replying on a 1Gig connection with a Protectli E3845 qp box. Because it can't get better then 500Mbps on pfSense with a vanilla install on it, I am out searching for something better. The only thing I have changed is the CPU offloading features in the Advanced network settings, and turning on AES-NI. My line is a Docsis 3.1 connection with 1Gbps x 35 Mbps. When I connect directly to the supplied modem with a gigabit NIC computer and no router, I can reach ~840Mbps download speeds and about 47Mbps upload.
This link has comments from Netgate (possibly directly from the owner) on this box: https://www.reddit.com/r/PFSENSE/comments/75k0mj/help_me_pick_out_my_new_firewall_hardware/
He even speaks to the NICs causing the quad core CPU to cut performance in half. Funny I chose this box because it had Intel NICs and AES and 2GHz cores thinking that was all I needed. Although I just bought the box about a month ago I can't return it.
If I buy a netgate brand box I am afraid I am going to need to spend around $2000 do get Gig throughput! I wanted low power consumption, but i guess that's a pipe dream until someone can help size these Netgate devices with Snort and other features running.
-
There's no need to spend thousands of dollars to get gigabit throughput. For instance, I have been running a Supermicro Xeon D-1518 based box on a symmetric gigabit connection for about 6 months now with no issues, and it readily passes gigabit speeds.
https://www.supermicro.com/products/system/1U/5018/SYS-5018D-FN8T.cfm
There are also faster/slower options and different form factors:
https://www.supermicro.com/products/embedded/embedded_server.cfmHonestly, this system is still overkill for what I need, but what really attracted to me was the number of high quality Intel network interfaces (4 x Intel 350, and 2 x Intel 210). I don't run OpenVPN (yet), but I imagine performance will be somewhat dampened given the lower clock speed (2.2GHz) of the D-1518. If OpenVPN is the top priority, one should look at CPU's with higher clock speed.
HTH
-
There's no need to spend thousands of dollars to get gigabit throughput. For instance, I have been running a Supermicro Xeon D-1518 based box on a symmetric gigabit connection for about 6 months now with no issues, and it readily passes gigabit speeds.
Are you using PPPoE? (ericseastrand & tman222)
https://www.supermicro.com/products/system/1U/5018/SYS-5018D-FN8T.cfm
Cool device, really! Did you install on an 2,5" SSD or M.2 SSD?
Honestly, this system is still overkill for what I need, but what really attracted to me was the number of high quality Intel network interfaces (4 x Intel 350, and 2 x Intel 210).
If enough RAM is inside, you will be able to set up anything needed turning it into nearly an UTM device.
I don't run OpenVPN (yet), but I imagine performance will be somewhat dampened given the lower clock speed (2.2GHz) of the D-1518. If OpenVPN is the top priority, one should look at CPU's with higher clock speed.
Also that can be tuned right a little bit;
- activating LZO compression
- activating UDP Fast I/O
- set the buffer to 2 MB try both lower and higher
- set up the mbuf size between 125000 and 1000000
- set the amount of num.queues on something between 1 and 4
For sure the best think would be owning a strong and high scaling CPU to get better OpenVPN throughput
but pfSense version 2.4.0 was changing and adding some new things so they use now LZO4 compressionIf there will be only the need to get 1GB/1GB Internet connection routed right, you can have luck and go with
$200 - $300 hardware often, but then also getting out the most from OpenVPN I personally see spending
more money. If the small SG-3100 will be able to route 1 GBit/s at the WAN and delivers nearly ~300 MBit/s
over IPSec, it would my way to walk on. And if there will be then at some day something that speeds up OpenVPN
in real or better then now, I would perhaps change back. -
I am definitely a fan of SuperMicro ever sense I bought an ASRock Rack board for a FreeNAS build and regretted it after numerous issues with the board, and an RMA, but the prices have always been steep.
This device definitely is superior to the Netgate SG-4860, but it isn't passively cooled. How loud is it during regular use?
List price for the SG-4860 is US$749 at store.netgate.com. The SuperMicro box is US$800 at Amazon, and you still need RAM. $60 for 8GB of Crucial ECC, also at Amazon, so that I can be just as overkill with my RAM as that XEON processor.
The SuperMicro is tempting, but can an SG-4860 with Snort and pfBlockerNG and Squid all running on it saturate a Gigabit line? I honestly would go for the Netgate just to be supporting the project that extra inch and to have hardware that will be focused on.
When I look at the CPU in the SG-4860 as compared to my own Protectli FW4A, I struggle to believe that it will be able to when the Protectli can't do more then ~500Mbps.
http://cpuboss.com/cpus/Intel-Atom-E3845-vs-Intel-Atom-C2558I need to look at TMan's second link a little more to see if there would be something a little cheaper that does almost what that awesome box can do.
-
You don't need much to route at gigabit speeds with no packages.
J3355 should do the trick while being silent, low power, no moving parts.
Use an intel NIC with however many ports you need. i3xx runs cooler than PRO/1000 and has VT-_.
-
@BlueKobold:
Are you using PPPoE? (ericseastrand & tman222)
My fiber connection doesn't use PPPoE, or if it does, it's all encapsulated into the AT&T "router-gateway" device that they gave me, such that I cannot have pfSense doing the authentication, and still need to use their provided box, at least to authenticate.
My understanding is that it uses some sort of certificate-based authentication, but one thing is for sure: If you plug directly into the ethernet jack on the optical network terminal, you don't get internet.
To bypass the "router-gateway", I need 3 ethernet jacks for: LAN, WAN (from the ONT), and a 3rd that is bridged with the WAN port, to let the "router-gateway" still exchange 802.1x traffic with AT&T so that they know I'm a legit subscriber and let me on the network.
My main concern is that this "bridging" is going to require better hardware, since it will then be having to decide what traffic goes where.
-
The SuperMicro is tempting, but can an SG-4860 with Snort and pfBlockerNG and Squid all running on it saturate a Gigabit line? I honestly would go for the Netgate just to be supporting the project that extra inch and to have hardware that will be focused on.
pfBlockerNG alone might be related to the amount of lists you were using and all stuff that comes besides with this
lists as much IP list downloading and so on, how more you select there, also more RAM will be needed then too!!!
With Squid and perhaps SquidGuard or lightSquid on top, this might be pointed to the art and wise you are running
using and/or configuring the Squid proxy as well. For Squid and Snort or Suricata I really think you may not able to
get out the most of all things, but then we are also very close or nearly a full featured UTM device, please don´t
forget this too.My fiber connection doesn't use PPPoE, or if it does, it's all encapsulated into the AT&T "router-gateway" device that they gave me, such that I cannot have pfSense doing the authentication, and still need to use their provided box, at least to authenticate.
Ok with port opening and forwarding you may be also able to use the pfSense firewall for terminating the VPN connection
their at the WAN interface directly I mean. Double NAT will also "eat" something around 3% - 5% from the whole
throughput too!My understanding is that it uses some sort of certificate-based authentication, but one thing is for sure: If you plug directly into the ethernet jack on the optical network terminal, you don't get internet.
Perhaps they have registered the MAC address from the WAN interface (NIC) but this could be work around then too
as I am informed you may be able to set up at the WAN interface (NIC) another MAC address for it to solve this point
right. But if there is a ONT or fiber modem into the game you must be using it (AT&T device)!To bypass the "router-gateway", I need 3 ethernet jacks for: LAN, WAN (from the ONT), and a 3rd that is bridged with the WAN port, to let the "router-gateway" still exchange 802.1x traffic with AT&T so that they know I'm a legit subscriber and let me on the network.
Again, if they (AT&T) are working together with certificates you must do so, if they use only the registered MAC address
from the WAN interface you could try out to "walk around" by spoofing this MAC address.My main concern is that this "bridging" is going to require better hardware, since it will then be having to decide what traffic goes where.
I wont use a bridge! There is a golden rule for that: "Route where you can, and only bridge if you must!"
So based on that I personally would at first find out what services you want to offer or use, because if this
AT&T device is a router it could be;- in front of the pfSense firewall without any issues, but not terminating the VPN directly there
- in front of the pfSense firewall and only some ports and protocols must be opened and forwarded to the pfSense firewall
- in front of the pfSense firewall and acting in the so called "bridge mode" as a pure modem without WiF and VOIP
capabilities, but this could be served over an external WiFi AP behind the pfSense firewall and a CISCO VOIP box
that is not so really hard to pay for at amazon, here in Germany it is able to get from ~20 - ~50 € used or new.
-
You can put the ATT gateway in IP Passthrough or DMZ+ depending on which model you have. Just turn off all routing and firewall functions of the gateway. At that point all you have to deal with from the gateway is its NAT table, the crappy models is like 2k I think, the newer ones are 8k+. No double NAT involved.
-
I wish it could be as simple as just putting the router in bridge mode. This device has no such capability. There is "DMZ+", but it still ends up being a double-nat setup, which still imposes that limit of like 2k concurrent connections.
The only way that I am aware of to completely bypass that "router-gateway" device's limited nat table is to have your own router connected directly to the ONT, and having your router pass the authentication traffic to the "router gateway". This way it can still authenticate with AT&T, but you don't have to use it for NAT.
For more info, check out these articles:
http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NAT-table-limits
https://strscrm.io/bypassing-gigapowers-provided-modem.html
https://community.ubnt.com/t5/EdgeMAX-Stories/Bypassing-AT-amp-T-Fiber-Gateway-with-Edgerouter-Lite-newbie/cns-p/1862846I've even tried buying static a block of IPs to set up a static route, but all it will let me do is assign one of those for DHCP use. The ability to set up a "cascaded router" is there, but it seems to be disabled on my device (typical AT&T).
I think I'm going to try starting small with a Celeron box, and if I end up needing more power, I can always put that to use elsewhere. I'll let you guys know where I end up.
-
Before you throw cash at it I would see if your can sweet talk an att rep into getting you a newer gateway.
I just had 100/100 fiber installed yesterday. Using IP passthrough my pfSense gets WAN IP and everything works as usual. I'm still limited to gateway NAT take but it's nearly 9k entries I think.
-
I've even tried buying static a block of IPs to set up a static route, but all it will let me do is assign one of those for DHCP use. The ability to set up a "cascaded router" is there, but it seems to be disabled on my device (typical AT&T).
What kind of gateway you have there from AT&T? (Vendor/model/model number)
I think I'm going to try starting small with a Celeron box, and if I end up needing more power, I can always put that to use elsewhere. I'll let you guys know where I end up.
Could also be nice, but please be ensure that it is coming together with AES-NI inside!
-
What kind of gateway you have there from AT&T? (Vendor/model/model number)
They gave me a Pace 5268AC-FXN. It looks identical to the Arris one, but I'm not sure if there are any underlying differences.
Could also be nice, but please be ensure that it is coming together with AES-NI inside!
I ended up going with a modern Celeron G3930, which has AES-NI. Right now it's connected behind that Pace box, so still double-nat, and getting ~600MBps both ways. Once I get some time to tinker, I will try bypassing the Pace box with that "forward the auth packets" hack, and connecting directly to the optical linkup.
I'm actually not 100% sure of the best way to apply that using pfSense, since the articles I found are for various linux-based routers. I'm sure with enough tinkering I can figure it out, but if anyone has any tips, I'd be happy to hear them.
-
All of the "solutions" to bypass att gateways are pretty hacky at best. Nothing is clean and/or reliable.
Honestly, if you set it.up right you don't double Nat, you get the wan IP to pfSense.
Unless you're actually hitting the limit of the gateways may table and it is causing you noticeable problems, there is no advantage to bypassing it. There are however quite a few disadvantages to the hacky bypassing solutions currently known.
-
Not pretty actual, but able to march without any dumping, magic in the middle and so on!
Would be my 1st choiceI ended up going with a modern Celeron G3930, which has AES-NI. Right now it's connected behind that Pace box, so still double-nat, and getting ~600MBps both ways. Once I get some time to tinker, I will try bypassing the Pace box with that "forward the auth packets" hack, and connecting directly to the optical linkup.
Up link and magic in the middle would be not my way.
I'm actually not 100% sure of the best way to apply that using pfSense, since the articles I found are for various linux-based routers. I'm sure with enough tinkering I can figure it out, but if anyone has any tips, I'd be happy to hear them.
In my eyes it might be a good sounding method to call the AT&T support and ask for another device that brings you
into the situation that you could set up your own device. Larger companies are surely not using that devices and
had also to set up their equipment working fine. Its a try out, but perhaps there is something able to realize. -
My fiber connection doesn't use PPPoE, or if it does, it's all encapsulated into the AT&T "router-gateway" device that they gave me, such that I cannot have pfSense doing the authentication, and still need to use their provided box, at least to authenticate.
My understanding is that it uses some sort of certificate-based authentication, but one thing is for sure: If you plug directly into the ethernet jack on the optical network terminal, you don't get internet.
To bypass the "router-gateway", I need 3 ethernet jacks for: LAN, WAN (from the ONT), and a 3rd that is bridged with the WAN port, to let the "router-gateway" still exchange 802.1x traffic with AT&T so that they know I'm a legit subscriber and let me on the network.
My main concern is that this "bridging" is going to require better hardware, since it will then be having to decide what traffic goes where.
You're right, AT&T uses 802.1X authentication, not PPPoE. Even DSL now uses 802.1X, locking users into the awful CPE that AT&T provides. I think the workaround you're after only applies to fiber connections, but that's something.
-
The Celeron box I built originally wasn't powerful enough to do full gigabit WAN, and was topping out around 600mbps, so I picked up a Dell PowerEdge T30 on sale at $329USD using coupon 329#T30 (might still work – go grab one while you can!) This new box sports an Intel Xeon E3-1225 @ 3.3GHz, with a Passmark score of 7783, whereas that Celeron G3930 scored only 3044.
I followed you guys' suggestions and just set everything up in "DMZ+" mode. There's still an extra unnecessary hop through the AT&T router-gateway, but at least now I can use UPNP with decent speeds and ping. I will probably get bored one day and try bypassing the RG box just for fun, but for now I'm very happy with my connection, and even happier to be back on pfSense!
Relevant links:
https://www.dealnews.com/Dell-Power-Edge-T30-Xeon-Quad-Tower-Server-for-329-free-shipping/2107012.html
http://www.dell.com/en-us/work/shop/dell-poweredge-servers/poweredge-t30-mini-tower-server/spd/poweredge-t30/pet30_12084_3 -
The Celeron box I built originally wasn't powerful enough to do full gigabit WAN, and was topping out around 600mbps, so I picked up a Dell PowerEdge T30 on sale at $329USD using coupon 329#T30 (might still work – go grab one while you can!) This new box sports an Intel Xeon E3-1225 @ 3.3GHz, with a Passmark score of 7783, whereas that Celeron G3930 scored only 3044.
I followed you guys' suggestions and just set everything up in "DMZ+" mode. There's still an extra unnecessary hop through the AT&T router-gateway, but at least now I can use UPNP with decent speeds and ping. I will probably get bored one day and try bypassing the RG box just for fun, but for now I'm very happy with my connection, and even happier to be back on pfSense!
Relevant links:
https://www.dealnews.com/Dell-Power-Edge-T30-Xeon-Quad-Tower-Server-for-329-free-shipping/2107012.html
http://www.dell.com/en-us/work/shop/dell-poweredge-servers/poweredge-t30-mini-tower-server/spd/poweredge-t30/pet30_12084_3This is good to know! So you’re basically saying all you did is to put your new box in the Gateway’s DMZ, and you were good to go? No extra setup? What version of pfSense are you running?
-
I get 1gb/1gb from wan to lan on ATT network which is bridged not routed. CPU barely breaks a sweat.
Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: NoIts really doesn't take that much. Just be sure your NICs are sitting in pcie ports.
The AES-NI functions matter alot when handling VPN traffic. I have never, not once ever hit a bottleneck due to lack of AES-NI on a system, but you would if you had two networks in the same city or state and were passing traffic over vpn between pfsense and a very fast computer if both ends were capable of gigabit speeds. Per core performance is what matters most.
Otherwise latency and bandwidth availability are more likely to limit your throughput than lack of AES-NI
AES-NI won't do jack to help with WAN to LAN performance for most traffic. Just the encrypted stuff.This crazy awesome state of the art rig cost $75 on newegg. Off-lease. Refurbished. I forget…
I am a fan of AES-NI, but having AES-NI doesn't mean you will have fast Lan to Wan performance. To ensure that, make sure your CPU features good old fashioned speed.
passmark benchmark for the E3-1225 mentioned above this comment is 5954. Thats why he gets good throughput. It can probably go alot faster than 1gb/1gb.
Note the single thread rating: Single Thread Rating: 1747passmark on the E7500 I'm running in Florida is only 1876 and handles gigabit traffic with ease.
Single Thread rating: Single Thread Rating: 1204A dual core machine with a very high single thread rating will likely outperform a 4 core or 8 core machine with a higher over all benchmark but lower per thread ratings in most cases.
Using that logic, I went here https://www.cpubenchmark.net/singleThread.html
Sorted by single thread performance on chips that also support AES-NI. The Intel Core i3-7350K Kaby Lake Dual-Core 4.2 GHz immediately stands out as dirt cheap and wicked fast.
-
So you’re basically saying all you did is to put your new box in the Gateway’s DMZ, and you were good to go? No extra setup?
Basically, yeah. I actually have static IPs, so I just went into the AT&T router setup and told it to give my pfSense box a static IP, and selected Firewall: Disabled, which automatically puts it in "DMZ+" mode.
The only weird thing I had to do was to set it up the LAN on 192.168.2.x (instead of the default 192.168.1.x). For some reason (I think because AT&T's device uses the 192.168.1.x range by default) I couldn't ping the pfSense box on the LAN (even thought I got an IP from DHCP). Another valid solution could have been to put AT&T's device on 192.168.0.x, but I foresaw this eventually confusing their support techs, and/or giving them a reason not to assist me.What version of pfSense are you running?
2.4.0-RELEASE (amd64)
I get 1gb/1gb from wan to lan on ATT network which is bridged not routed. CPU barely breaks a sweat.
Are you on fiber-to-the-home by chance? If so, did you use the "pass the authentication over a bridge" hack described here: https://strscrm.io/bypassing-gigapowers-provided-modem.html?
A dual core machine with a very high single thread rating will likely outperform a 4 core or 8 core machine with a higher over all benchmark but lower per thread ratings in most cases.
…
passmark on the E7500 I'm running in Florida is only 1876 and handles gigabit traffic with ease.
Single Thread rating: Single Thread Rating: 1204Now I'm wondering why that Celeron box I built didn't perform despite having a single-thread rating of 1659. Maybe I was just testing it at a bad time, or against a slow/distant speedtest server. Who knows…
In the end it all works out though: My parents will get a new Windows PC, and I'll probably end up virtualizing this Xeon box and consolidating several other power sucking devices into one.