I bought a Supermicro 5018D-FN8T: The Chronicles [Edited Title]

[moscato359]

I got the base box from neweggbusiness for 799 usd (I contacted a rep there)

They didn't actually even sell the item normally, except marketplace, but they special ordered one from supermicro for me. It's bizarre what they'll do if you ask.

Got 97 usd samsung evo 850 250GB, and 8GB of ddr4 (2x4) ecc (I forget what was paid)

[Guest]

@moscato359:

I got the base box from neweggbusiness for 799 usd (I contacted a rep there)

They didn't actually even sell the item normally, except marketplace, but they special ordered one from supermicro for me. It's bizarre what they'll do if you ask.

Got 97 usd samsung evo 850 250GB, and 8GB of ddr4 (2x4) ecc (I forget what was paid)

Cool, this might be a Long time running pfSense box mith much ahead space for other things!
If you got the 10 GBit/s Switches please post here a following comment about the performance, comp ability,
throughput and so on, I will be very interested on this. For sure pfSense in the version 2.4 will not be ready now
for the usage in production networks but on the other side if you only use it as a firewall without any packets it
might be a chance to get better driver und hardware support, or am I wrong with that meaning?

[VincentV]

I've just completed some performance testing (using NTTtcp) on the units I have using pfSense 2.3.3.
For some odd reason now the media speed is being detected properly, so that might have been just some random quirk in my previous testing.

If you are going to be routing traffic between the 10G interfaces, I highly suggest you turn off the power management features in the BIOS. HyperThreading does help though, with HT off I lose about 200 MB/s throughput.

In my testing that was the difference between ~480-550 MB/s  to ~750-980 MB/s ( 4 threads, single direction) between these two interfaces.  Single thread I got ~510 MB/s, which is more than sufficient for my usage.
Of course the hosts generating the traffic were easily able to reach  1100+ MB/s when directly connected.

Power usage when pushing 800+ MB/s of traffic was around 44W at the wall.
Disabling the power efficiency stuff in BIOS had a small impact on idle power: from ~29W  to 33W.

I have a few more days before I start configuring it to replace my current aging pfSense router, if there's any test I could run, let me know!


[moscato359]

I'm actually having some trouble with ipv6 on this box

I have my 2 boxes in HA

supermicro box
netgate box

Their XML configs are synced

Everything works great… except ipv6

If I setup dhcpv6 on the supermicro on wan, and tracking on lan, lan never gets an IPv6 address

If I setup dhcpv6 on the netgate on wan, and tracking on lan, lan gets an IPv6 address

Haven't been able to figure out why

The configs look identical to me

Maybe its a driver issue?

This is 2.3.3u1

[moscato359]

One thing I found odd:

My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"

The entire line was missing from my firewall.

I had to configure it manually to use the AES-NI hardware crypto engine

[VAMike]

@moscato359:

One thing I found odd:

My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"

The entire line was missing from my firewall.

I had to configure it manually to use the AES-NI hardware crypto engine

If you're using ipsec you just sped things up, if you're using openvpn you just showed things down.

[moscato359]

@VAMike:

@moscato359:

One thing I found odd:

My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"

The entire line was missing from my firewall.

I had to configure it manually to use the AES-NI hardware crypto engine

If you're using ipsec you just sped things up, if you're using openvpn you just showed things down.

Interesting.

That explains the default off… but doesn't explain why it was turned on, on the netgate image

So goofy

[tman222]

I know this thread is a few months old, but I thought weigh in quick and give my thoughts as well as I have been using this exact server with pfSense for the last 3 - 4 months on a symmetric 1Gbit internet connection.  Overall, it has performed wonderfully and was very easy to setup initially.  Everything worked right out of the box and there were no major hardware or software issues (I started with pfSense 2.3.3 and now using the latest 2.3.4).  The only thing I could not get to work during the install was a M.2 SSD, but this may have been due to the lack of support in the version of FreeBSD that 2.3.3 is using.  A normal (SATA) SSD worked just fine and pfSense installed without any issues.  The raw speed of the box is probably a little overkill for the current size of my network, but it's always good to future proof a bit, and why not go with a little extra horsepower when the power consumption is nice and low (only 35W TDP on the CPU is quite impressive!)?  The only issue I ran into was some instability under high load when traffic shaping is enabled.  However, a workaround (i.e. manually changing the igb queues to 1) seems to have fixed this problem.  See:

https://forum.pfsense.org/index.php?topic=132345.0

Overall, I have been quite impressed with this box thus far.

[moscato359]

I actually forgot about this thread. Your response sent me an email.

It worked great until I was laid off a couple months later.

It's a pretty good server.

[chrcoluk]

if you want to have least impact on performance with best power efficiency I wouldnt let the cpu go to idle clocks, but keep c-states enabled, c-states save's far more power than eist and c1,c2 are both very quick and cheap for performance. c3 is a jump up from c2, but a trick is to only enable c3 on say half the cores, so half the cores will still respond instantly for interactive stuff, whilst c3 will still wake up quick enough to deal with loads that need all cores.