First pfSense box - Xeon build

whosmatt

Also picked up an arris sb6190

Might want to see this:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mistake

toyebox

@toyebox:

Also picked up an arris sb6190

Might want to see this:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mistake

Thank you ! I have canceled the order for that. I assumed that all the great reviews of the smaller model that this one would accel. Do you have a suggestion?

@BlueKobold:

E3 1265Lv2

Get it! It scales from 2,5GHz to 3,5GHz and has 4C/8T (HT) and AES-NI on top too.

Silverstone pt13D case (looks fantastic!)

If this is the Silverstone case where you want to put outthe extra NIC with 2 or 4 Ports?

I don't need them to be honest. That's why I picked of the edgeswitch. What benefit is there to having them? Is there a specific configuration that you are referring to?

Thanks everyone for the knowledge and opinions! I love it. Keep it coming

toyebox

Can someone please comment on my above comment? About not needing the extra ports. My whole system was designed around using Tue edgeswitch in it. Are there major benefits to hook up an AP or PC directly to a pfSense box?

whosmatt

@toyebox:

Thank you ! I have canceled the order for that. I assumed that all the great reviews of the smaller model that this one would accel. Do you have a suggestion?

I don't, off the top of my head. I just switched from cable back to vdsl (slower speeds, but unlimited data) and have sidelined my cable modem. I was running a Zoom 5370 (16x4) that worked like a champ, but you'll have to do your own research for 24 or 32 channel downstream DOCSIS 3 modems. Just wanted to make you aware of the ongoing issue. It's not just the 6190; any modem with the Intel Puma 6 chipset is affected. Look for one with a Broadcom chipset if you need that many channels.

toyebox

So all my parts have arrived . going to be building my new box soon here.. Going to try with the e3 1220L for shits and giggles. I want to benchmark it with a VPN running, snort and squid I think. Can anyone point me in the direction of a good guide on accurately testing throughput?

datum

I am rather curious because i am also planning on a build based on a xeon.
Please post your progress.Thanks

toyebox

So the box is up, configured and paired with an 8 port Edgeswitch. I have an openvpn server running already, snort installed and i have ran some tests. Everything seems to be fantastic from what i can see. throughput of 940MB/s with snort and all. Haven't tested through a VPN yet though. On a side note, i have link aggregation setup from a FreeNAS box and my Edgeswitch and i was running some tests on that as well.. here are my results.

Interface Traffic Peak Total

lagg0 in 232.836 MB/s 233.663 MB/s 80.145 GB
out 5.025 MB/s 121.171 MB/s 9.453 GB

em1 in 115.722 MB/s 116.433 MB/s 50.142 GB
out 3.949 MB/s 22.756 MB/s 3.619 GB

em0 in 117.335 MB/s 117.338 MB/s 30.032 GB
out 1.075 MB/s 80.398 MB/s 5.835 GB

Fully saturated 100% across both connections.

obviously, this isn't even doing anything to pfSense though, as it's running all through locally via the switch. Due to only having two NICS on the pfSense box, the only way i can test my throughput of pfSense is to do a makeshift configuration with a computer connected to the WAN and another connected to the LAN.. unless im missing something :)

Question/verification.. Snort only watches the WAN, correct?

datum

Great results, very good indeed.
Thanks for you update Toyebox, for a Processor Base Frequency of 2.50 GHz thats very good.

I am thinking of a 1151 cpu for my rig.

The edgeswitch surely plays its part, but with your setup on full power, what results do you get on terms of cpu usage?
Only one core tops out, several cores, and with snort you really get all your bandwithd ?

I am asking because if i can save up money on a lower mhz Xeon, that would be super.No need to go too overkill.

If a 1220 at 3ghz does the job no need to buy a 1240.

Please keep us informed, thanks.

toyebox

@datum:

Great results, very good indeed.
Thanks for you update Toyebox, for a Processor Base Frequency of 2.50 GHz thats very good.

I am thinking of a 1151 cpu for my rig.

The edgeswitch surely plays its part, but with your setup on full power, what results do you get on terms of cpu usage?
Only one core tops out, several cores, and with snort you really get all your bandwithd ?

I am asking because if i can save up money on a lower mhz Xeon, that would be super.No need to go too overkill.

If a 1220 at 3ghz does the job no need to buy a 1240.

Please keep us informed, thanks.

My full setup currently is:

i7 3770S
16GB nonECC memory
120GB msata SSD (samsung)
Intel BOXDQ77KB Mobo (using the dual onboard Intel NICs)

I ran some Iperf tests. The setup looks like this:

modem
|
|
|
v
random router (pfSense is DMZ)
|
|
v
WindowsPC (Iperf Server) –-------> switch ---------> pfSense ---------> FreeNAS box (Iperf client)

Reasoning for the odd setup is due to the fact that i only have two NIC's in my pfSense box and i only have a 100Mbit/sec connection.. I wanted to really push it using iperf so i put a computer on the WAN side of the pfSense box.

The commands i used are:

Server: iperf -s -u -i 1 -B 192.168.1.10 -p 7001
Client: iperf -c 192.168.1.10 -B 192.168.0.5 -t 99999999 -u -i 1 -p 7001 -b 1000M -l 1250 -S 0xA0

-c 192.168.1.10 = server ip
-B 192.168.0.5 = the clients IP
-t 99999999 = run infinitely
-u = use UDP
-i 1 = 1 second between bandwidth reports
-p 7001 = port used
-b 1000M = size of bandwidth to send per second
-l 1250 = length of buffer
-S 0xA0 = Type of service to report (means Critical)

I tried 1000M first. Results:

[ 3] local 192.168.1.10 port 7001 connected with 192.168.1.21 port 62141
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 1.0 sec 113 MBytes 944 Mbits/sec 0.017 ms 3488/97866 (3.6%)
[ 3] 1.0- 2.0 sec 112 MBytes 943 Mbits/sec 0.017 ms 5707/100023 (5.7%)
[ 3] 2.0- 3.0 sec 112 MBytes 943 Mbits/sec 0.018 ms 5638/99980 (5.6%)
[ 3] 3.0- 4.0 sec 112 MBytes 943 Mbits/sec 0.017 ms 5651/99993 (5.7%)
[ 3] 4.0- 5.0 sec 112 MBytes 943 Mbits/sec 0.018 ms 5638/99976 (5.6%)
[ 3] 5.0- 6.0 sec 112 MBytes 943 Mbits/sec 0.018 ms 5638/99976 (5.6%)

About average of 5.5% loss which i believe to be pretty accurate given the max bandwidth output through a gigabit LAN is about 940Mbits/sec.

Then i tried to lower it down to 900 Mbits/sec

[ 3] local 192.168.1.10 port 7001 connected with 192.168.1.21 port 65495
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 1.0 sec 107 MBytes 900 Mbits/sec 0.018 ms 0/90041 (0%)
[ 3] 1.0- 2.0 sec 107 MBytes 900 Mbits/sec 0.018 ms 0/89995 (0%)
[ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec 0.015 ms 0/90008 (0%)
[ 3] 3.0- 4.0 sec 107 MBytes 900 Mbits/sec 0.018 ms 0/90004 (0%)
[ 3] 4.0- 5.0 sec 107 MBytes 900 Mbits/sec 0.018 ms 0/90003 (0%)
[ 3] 5.0- 6.0 sec 107 MBytes 900 Mbits/sec 0.019 ms 0/89995 (0%)
[ 3] 6.0- 7.0 sec 107 MBytes 900 Mbits/sec 0.019 ms 0/90006 (0%)
[ 3] 7.0- 8.0 sec 107 MBytes 900 Mbits/sec 0.019 ms 0/90001 (0%)

as expected, zero loss.

I am not totally sure how much stress this is putting on the system, even with snort enabled.. I am still very new at pfSense, so i am not sure if snort puts a heavy load on the system just from being enabled, or when there is a mass amount of traffic with a small window size? If anyone can clear this up for me, and give me an idea of how to correctly test the performance (if the above is incorrect) , i would appreciate it. As for now, i am very satisfied with the throughput of this system. If anyone else wants to see any other tests, feel free to ask and i will do what i can. I did try some torrenting with an ubuntu CD as the media, and easily maxed out at my max internet speed of 100Mbits/sec (hopefully 1Gbits/sec soon!!)

Thanks for the advice and help so far!

VAMike

I thought the idea here was that you wanted something to support gigabit vpn and maybe snort. It's complete overkill for just firewalling (if you couldn't run iperf through at 1000Mbps it would be a real problem). You can test the VPN performance to get a better idea of how the CPU does. As for snort, it's mostly single thread and if it can't keep up it'll just drop traffic (won't affect the traffic going through). Testing that is a lot harder, and the performance is heavily dependent on the traffic and the signatures (you can't test just by running iperf through it).

toyebox

@VAMike:

I thought the idea here was that you wanted something to support gigabit vpn and maybe snort. It's complete overkill for just firewalling (if you couldn't run iperf through at 1000Mbps it would be a real problem). You can test the VPN performance to get a better idea of how the CPU does. As for snort, it's mostly single thread and if it can't keep up it'll just drop traffic (won't affect the traffic going through). Testing that is a lot harder, and the performance is heavily dependent on the traffic and the signatures (you can't test just by running iperf through it).

My apologies, i do plan on using it for VPN. I will try running a VPN through it now. Do you have suggestions on what tests to perform? Like i said, i am restricted by my actual line connection at the current time, so thats why i defaulted to iperf. I would happily take suggestions of what tests to perform to get a good idea of my performance. Like i said, i am fairly new at this. Networking isn't my main expertise.. programming is; but i would like to get better!

corvey

@toyebox:

-I am having a very hard time finding a capable Mobo in mini ITX form.. Any suggestions would be greatly appreciated. This is the only one i could find. http://ark.intel.com/products/59046/Intel-Desktop-Board-DQ77KB

Would you recommend getting a i350 pciE? I would much rather just use the dual intel NIC's if i can, but only if they will be efficient enough to handle the load. I have a feeling adding the Ethernet card will also cause problems with the small form factor.

I bought a DQ77KB and a 1265L V2 over a year ago and runs it under VMware's ESXi. It runs pfsense on a dedicated msata and 3 operating systems on separate SSD. All while using a Chinese knockoff i350 quad nic and it's been running 100% solid the entire time! I can't recommend this system setup enough. It's been a marvel of reliability.

toyebox

@corvey:

@toyebox:

-I am having a very hard time finding a capable Mobo in mini ITX form.. Any suggestions would be greatly appreciated. This is the only one i could find. http://ark.intel.com/products/59046/Intel-Desktop-Board-DQ77KB

Would you recommend getting a i350 pciE? I would much rather just use the dual intel NIC's if i can, but only if they will be efficient enough to handle the load. I have a feeling adding the Ethernet card will also cause problems with the small form factor.

I bought a DQ77KB and a 1265L V2 over a year ago and runs it under VMware's ESXi. It runs pfsense on a dedicated msata and 3 operating systems on separate SSD. All while using a Chinese knockoff i350 quad nic and it's been running 100% solid the entire time! I can't recommend this system setup enough. It's been a marvel of reliability.

Very nice!! I honestly have never messed with ESXi. Maybe I will have to give it a try.

VAMike

@toyebox:

My apologies, i do plan on using it for VPN. I will try running a VPN through it now. Do you have suggestions on what tests to perform?

Do exactly what you did, except with a vpn server set up in the middle?

toyebox

@VAMike:

@toyebox:

My apologies, i do plan on using it for VPN. I will try running a VPN through it now. Do you have suggestions on what tests to perform?

Do exactly what you did, except with a vpn server set up in the middle?

Okay! Will do!