I bought a Supermicro 5018D-FN8T: The Chronicles [Edited Title]
-
Wow, I literally just unpacked this same Supermicro server I received this morning!
PfSense was the first thing I wanted to test on it and I'm really happy to say everything appears to be in order.
The 10G ports are readily usable, although it seems to be unable to report the connection speed. My ubiquity switch does report 10G Full duplex connection though.Power consumption is at ~30-35W measured using an el-cheapo watt-meter when the system is idling, but I haven't done any bios tweaking to try to bring that down yet.
Now I'm off to test Freenas 9.10!
Were you using any pfsense based cpu frequency scaling? 30-35w idle is pretty high
I was also looking at getting ubiquiti switches at work to replace our cisco sg-200s
Are you using ES or UN switches? How do you like them?
-
I installed the current snapshot of 2.4 and fooled around a little bit.
The connection speed on the 10G port is now being reported correctly.
After updating to BIOS 1.0b the fan speeds seem to be lower and my guess is that has contributed to lower power use. It is now at 26-27W while idle.
Enabling/disabling PowerD in the advanced settings doesn't seem to have any effect on power usage.The switch I'm using is the Ubiquity ES-16-SG. (much better value for money than the Dell X4012!)
So far so good, granted my usage is pretty basic, mostly I just need VLANs. The web interface is standard Ubiquity fare, if you've used their APs before you know what I mean. There is CLI available via SSH or console.
-
I installed the current snapshot of 2.4 and fooled around a little bit.
The connection speed on the 10G port is now being reported correctly.
After updating to BIOS 1.0b the fan speeds seem to be lower and my guess is that has contributed to lower power use. It is now at 26-27W while idle.
Enabling/disabling PowerD in the advanced settings doesn't seem to have any effect on power usage.The switch I'm using is the Ubiquity ES-16-SG. (much better value for money than the Dell X4012!)
So far so good, granted my usage is pretty basic, mostly I just need VLANs. The web interface is standard Ubiquity fare, if you've used their APs before you know what I mean. There is CLI available via SSH or console.Thank you for testing 2.4 for me (or atleast, testing it, and telling me the results). Knowing that it works better on 2.4 is exactly what I needed to know
You're my hero.
Answering all the questions I need, when I've been scared to order one for work.
What made you go with ES over UN?
I've been considering UN products, because I also have 6 unifi APs. The integration would be nice.
If you can make a solid argument for ES over UN, I'll consider it.
I need like, 2 (48p + 2 10gbe) switches, and a couple 16 port ones for specific areas.
-
So I got the box in
First issue I had with 2.3.3u0 is that igb2-5 aren't working
igb0 and igb1 are workingTrying an update to see if it works
If it doesn't, I'll have to try 2.4, which I really don't want to use in production
-
Thanks @VincentV for the unpacking and the screenshots! This units rocks well as I see it right and one of them would be
my next pfSense if I see it right now! Supermicro SYS-300E-8D or Supermicro SYS-300E-8D I am watching over.If the latest BIOS is on the hardware I would do the following now;
- enabling Hyper threading in the BIOS!
- Disable the IPMI port to be the fall back WAN port!
- Enable the TRIM support in pfSense
- Enable the PowerD (hi adaptive) mode based on the information in the dashboard (800MHz - 2200MHz) then it
will scaling up and down likes it is needed by or for the OS. - high up the mbuf size to 1000000 beased on the amount of RAM inside of that machine
Could be a really nice pfSense firewall with much abilities.
-
Thanks for the tips.
I got the network adapters to show as link-up by assigning IPs. For some reason, without IPs assigned, 0 and 1 could show as link up, but 2 - 5 needed IPs
I haven't tested the SFP+, as I don't have other SFP+ hardware… yet
I plan on getting some Ubiquiti switches in a couple months
-
I was hoping to get something with 10gbe for work, but the prices on official 10gbe hardware are too high.
The D-Link DGS1510-20 would be a nice budget switch with 2 x SFP+ ports.
10gbe would let me vlan to my hearts content without any concern for bandwidth[end original text]
This might be better then for sure, and you don´t need to fiddle something together such as a LACP based or static LAG.
And the throughput is mostly not protocol independent and would be pending between 2 GBit/s and 4 GBit/s in real.
If you take IPerf or NetIO for doing speed tests it might be between 8 GBit/s and 9,5 GBit/s but this is then protocol
independent measured and more of a theoretical nature.If you are setting up a DMZ and a LAN area it might be better to take inside of the DMZ a Layer2 "only" Switch and inside of the
LAN area a Layer3 Switch. If you get switches with 10 GBit/s uplinks and 1 GBit/s Ports it might be a fine gain for the entire network
throughput.This is a xeon-d machine similar to the one that's in the 2500$ official pfsense hardware.
Bare bone ~990 Euro
2 x 4 GB RAM ~100 Euro
120 mSATA ~60 EuroInstead of the xeon-d 1541 cpu, it has 1518, and 2 additional nic ports.
Not really and only as I see it right, the cost itself is nice cheaper but the tunings and pimps done by the
developers might be making the original more interesting for business clients.Would there be any issues with the chipset, nics, or any other drivers with working with pfsense?
No, I think this example shown by @VincentV shows it right up that the entire hardware will be supported and in
version 2.4 it will be a better support for sure likes the version now. Newer or the newest hardware will mostly not
really at the time supported in BSD systems likes under MS Windows or Linux. -
I got the base box from neweggbusiness for 799 usd (I contacted a rep there)
They didn't actually even sell the item normally, except marketplace, but they special ordered one from supermicro for me. It's bizarre what they'll do if you ask.
Got 97 usd samsung evo 850 250GB, and 8GB of ddr4 (2x4) ecc (I forget what was paid)
-
I got the base box from neweggbusiness for 799 usd (I contacted a rep there)
They didn't actually even sell the item normally, except marketplace, but they special ordered one from supermicro for me. It's bizarre what they'll do if you ask.
Got 97 usd samsung evo 850 250GB, and 8GB of ddr4 (2x4) ecc (I forget what was paid)
Cool, this might be a Long time running pfSense box mith much ahead space for other things!
If you got the 10 GBit/s Switches please post here a following comment about the performance, comp ability,
throughput and so on, I will be very interested on this. For sure pfSense in the version 2.4 will not be ready now
for the usage in production networks but on the other side if you only use it as a firewall without any packets it
might be a chance to get better driver und hardware support, or am I wrong with that meaning? -
I've just completed some performance testing (using NTTtcp) on the units I have using pfSense 2.3.3.
For some odd reason now the media speed is being detected properly, so that might have been just some random quirk in my previous testing.If you are going to be routing traffic between the 10G interfaces, I highly suggest you turn off the power management features in the BIOS. HyperThreading does help though, with HT off I lose about 200 MB/s throughput.
In my testing that was the difference between ~480-550 MB/s to ~750-980 MB/s ( 4 threads, single direction) between these two interfaces. Single thread I got ~510 MB/s, which is more than sufficient for my usage.
Of course the hosts generating the traffic were easily able to reach 1100+ MB/s when directly connected.Power usage when pushing 800+ MB/s of traffic was around 44W at the wall.
Disabling the power efficiency stuff in BIOS had a small impact on idle power: from ~29W to 33W.I have a few more days before I start configuring it to replace my current aging pfSense router, if there's any test I could run, let me know!
 -
I'm actually having some trouble with ipv6 on this box
I have my 2 boxes in HA
supermicro box
netgate boxTheir XML configs are synced
Everything works great… except ipv6
If I setup dhcpv6 on the supermicro on wan, and tracking on lan, lan never gets an IPv6 address
If I setup dhcpv6 on the netgate on wan, and tracking on lan, lan gets an IPv6 address
Haven't been able to figure out why
The configs look identical to me
Maybe its a driver issue?
This is 2.3.3u1
-
One thing I found odd:
My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"
The entire line was missing from my firewall.
I had to configure it manually to use the AES-NI hardware crypto engine
-
One thing I found odd:
My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"
The entire line was missing from my firewall.
I had to configure it manually to use the AES-NI hardware crypto engine
If you're using ipsec you just sped things up, if you're using openvpn you just showed things down.
-
One thing I found odd:
My netgate box says this "Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM"
The entire line was missing from my firewall.
I had to configure it manually to use the AES-NI hardware crypto engine
If you're using ipsec you just sped things up, if you're using openvpn you just showed things down.
Interesting.
That explains the default off… but doesn't explain why it was turned on, on the netgate image
So goofy
-
I know this thread is a few months old, but I thought weigh in quick and give my thoughts as well as I have been using this exact server with pfSense for the last 3 - 4 months on a symmetric 1Gbit internet connection. Overall, it has performed wonderfully and was very easy to setup initially. Everything worked right out of the box and there were no major hardware or software issues (I started with pfSense 2.3.3 and now using the latest 2.3.4). The only thing I could not get to work during the install was a M.2 SSD, but this may have been due to the lack of support in the version of FreeBSD that 2.3.3 is using. A normal (SATA) SSD worked just fine and pfSense installed without any issues. The raw speed of the box is probably a little overkill for the current size of my network, but it's always good to future proof a bit, and why not go with a little extra horsepower when the power consumption is nice and low (only 35W TDP on the CPU is quite impressive!)? The only issue I ran into was some instability under high load when traffic shaping is enabled. However, a workaround (i.e. manually changing the igb queues to 1) seems to have fixed this problem. See:
https://forum.pfsense.org/index.php?topic=132345.0
Overall, I have been quite impressed with this box thus far.
-
I actually forgot about this thread. Your response sent me an email.
It worked great until I was laid off a couple months later.
It's a pretty good server.
-
if you want to have least impact on performance with best power efficiency I wouldnt let the cpu go to idle clocks, but keep c-states enabled, c-states save's far more power than eist and c1,c2 are both very quick and cheap for performance. c3 is a jump up from c2, but a trick is to only enable c3 on say half the cores, so half the cores will still respond instantly for interactive stuff, whilst c3 will still wake up quick enough to deal with loads that need all cores.