Supermicro Xeon D X10SDV-4C+-TP4F with 10Gb SFP+ pfSense compatibility?
-
I definitely appreciate the heads up on a potentially troublesome problem. Looking at the pfSense bug report it may have been addressed in FreeBSD12?
How would I know if this would affect me or not, I think everybody eventually uses VLAN? I don't plan to use link aggregation, though. In fact my plan is to use one 10Gb SFP+ to the LAN switch, and one 1GB I210 Intel NIC to the WAN.
-
@lifespeed yeah I think it will be fixed in pfSense 2.5 probably some way out. I'm not sure outside of testing it. I know it has the issue when in a LAGG and it's related to VLAN updates but don;t know if using single links if it would be a problem
-
@Hass, I definitely appreciate the heads-up. I'm no IT expert, this is a bit of a stretch for me. But it is clear my 10-year-old "prosumer" router has to go, and I wanted to do something with some growing room.
The error you describe could really stump a novice like myself. If I were to encounter it in my configuration . . .
-
Hi @lifespeed - I'm using the Supermicro 5018D-F8NT system with pfSense, which is based on this board:
https://www.supermicro.com/en/products/motherboard/X10SDV-TP8F
In terms of specifications, this board looks very similar to the one you linked to. In my case, the onboard SFP+ interfaces are actually using the ix (ixgbe) driver in pfSense / FreeBSD as opposed to the ixl drivers.
https://www.freebsd.org/cgi/man.cgi?query=ix&apropos=0&sektion=4&manpath=FreeBSD+11.3-RELEASE+and+Ports&arch=default&format=html
I'm thinking this may be the case for the X10SDV-4C+-TP4F as well. Overall, this setup has been flawless with the exception of this issue (which has only been occurring recently):
https://forum.netgate.com/topic/147301/sfp-ixgbe-network-interface-won-t-come-back-up
Hopefully this will resolved in the next version of pfSense and newer version of FreeBSD.
Hope this helps.
-
@tman222, thanks for sharing your experiences. It sounds a little flaky, but not insurmountable. It sounds like a few people are looking forward to some fixes in the new version, which is not the greatest position to be in.
I'm not sure there are many other good options for 10Gb SFP router hardware.
-
@lifespeed said in Supermicro Xeon D X10SDV-4C+-TP4F with 10Gb SFP+ pfSense compatibility?:
@tman222, thanks for sharing your experiences. It sounds a little flaky, but not insurmountable. It sounds like a few people are looking forward to some fixes in the new version, which is not the greatest position to be in.
I'm not sure there are many other good options for 10Gb SFP router hardware.
Hi @lifespeed - the other option would be to use an SFP+ add-on card. I have used both the Chelsio T520 and T540 SFP+ cards (in fact the T540 is currently installed) and these have been rock solid.
-
Hello
I am looking at this board for a new pfsense server.
Can anyone tell me if all issues were sorted out on pfsense 2.5 for this motherboard?
It really looks like the perfect solution, but I am concerned about the VLAN issues mentioned here. The upgrade to FreeBSD 12 should have fixed it?Thanks
-
@pj I've been using X11SDV-4C-TP8F with 10Gb fiber SFP+ on the LAN side, 1Gb copper Ethernet on the WAN side. I haven't used LAG or VLAN. It works well and has great performance. However, I did use slower, quieter fans on the exhaust, 40mm, 7 total. The passively-cooled CPU really wasn't cutting it, so I found a hard-to-find heatsink compatible with the motherboard that mounts an 80mm fan, cut a hole in the cover, and poked the CPU fan through the top of the 1U rack-mount chassis. Cool and quiet. Really, if you're not willing to put up with screamer fans, this is the only reasonable option.
The performance of the hardware and pFsense has been great. I'm still on 2.4.5, haven't upgraded to 2.5 yet.
-
@lifespeed That is good news.
Those turbine fans are really noisy. I have two older model Supermicro switches that are also too noisy to be used inside of an office, but I have to say that I have not had a single faulty fan on any of my Supermicro servers yet, so I understand why they use turbine fans in everything.
I plan on mounting the server inside of a data center, so it can make as much noise as it wants to.
Thank you for the feedback, and the detailed post.
-
@lifespeed would you be able to share what SFP+ transceiver you are using and a picture of your case mod? It would be very much appreciated.
I'm just building up an X11SDV system in a 1U supermicro case, and the passive cooler does get very warm. I've had one thermal shutdown today before I threw on a bigger fan.
This is a replacement for a pfSense system which I've been running for a few years. Ultimately moving from VDSL to fibre in a few weeks, when I have the new system tuned.
Thanks in advance... Aidan
-
@aidanic I'll see what I can dig up this evening. The heatsink w. fan is a specific Supermicro part, of course.
-
@lifespeed Thanks...
This is my summary so far... like yourself, I'm doing a major bit of future-proofing with this pfSense setup.
I have the X11SDV in a SuperMicro CSE-505-203B 1U case. The case came with few accessories, so I'm waiting on the official 40mm fans (FAN-0065L4). Cable management is painful, even with the M.2 SSD I'm using.
RAM has been a challenge too. After two different sets of RAM failed, I returned the motherboard. It came back two weeks later with "no problems". I then got Samsung RAM which is now working fine - Samsung 16 GB reg. ECC DDR4-3200 DIMM M393A2K43DB3-CWE. I had not expected the motherboard to be quite so picky about RAM modules.
The stock 2.5 pfSense has come up fine with the four 1G ports. When I get the rest of the case assembled, this will be where I start, replacing my existing pfSense setup.
The SFP+ ports are more troublesome. I have a few (random) SFP LC 1G transceivers, and they are returning unqualified sfp+ module detected when plugged in. My single 10G Ethernet transceiver appears OK. I'm waiting on some "Intel" transceivers from FS later this week. While I'm preparing for 10G, all my existing infrastructure is 1G, and my testing opportunities are limited.
Have you done anything with ixgbe settings in loader.conf? Or is it that your transceivers were already on the magic whitelist in the intel firmware?
Many thanks,
Aidan
-
@aidanic I had no difficulty with transceivers, although I did take care to use a model that was compatible with my Ubiquiti US-16-XG switch. Apparently they work fine in the X11SDV as well.
I'm afraid I forgot about the photos and part numbers last night, will try again tonight if I get through taxes.
-
@lifespeed The "Intel" transceivers arrived this morning from FS, and the 10G plug in fine with no error messages. With no other 10G infrastructure, I can't test those end-to-end, although they are probably fine.
The new 1G transceivers also report "unqualified" - and clearly any random mix/match of 10G/1G does not work. I'll go back to look at this after I've sorted out the cooling side of things.
-
@aidanic The part number for the heat sink fan is SNK-C0111AP4L. You'll have to cut a square whole in the 1U lid, I sealed mine with natural rubber electrical tape. Also I recommend using Noctua 40mm fans mounted with silicone nails to quiet it down, but you'll need enough to cover the whole rear panel. Also use a Noctua 60mm(?) heat sink fan, as the OEM Supermicro is a screamer as well. Might be 7 rear fans if I remember correctly. They don't move as much air as the screamers, but it does cool adequately in conjunction with the active HSF.
I actually hacked off the "extra" power supply wires to make room for airflow and wire dressing. I'm not putting any hard drives in this box, so was able to get away with fewer supply wires.
-
@aidanic here's a photo of my whole rack, all I can find at the moment. If you zoom in you can see the HSF sticking out like a supercharger through the hood of an old musclecar. This is before I swapped from the Supermicro HSF screamer fan to Noctua, and removed the tape blocking part of the airflow in the front.
CPU and case fans are all controlled by the motherboard, you can log into the IPMI interface and configure fan control for good cooling with low noise.
-
@lifespeed Thanks - I can see such a casemod coming in my future. I have an 80mm fan just sitting on the stock heatsink with the case off, which is keeping the system "cool enough" for my initial setup and testing.
There's a bigger thread over on servethehome around the cooling of the X10SDV and X11SDV motherboards including various 3D printed items Cooling the ... X11SDV ... It is worth looking at for ideas around cooling.
I'm not the only one having thermal shutdown issues with the X11SDV without enhanced cooling.
I ran my pfSense configuration on the X11SDV yesterday for a bit without issue using the 1G Ethernet ports. SFP and fibre can wait.
-
@aidanic The thread you linked is where I found out about the Supermicro HSF. Hopefully you can still buy one, as far as I know it is the only practical solution for a close-to-1U implementation. Good luck taming the screamer fans, at least if noise is a concern. Mine is acceptably quiet, although obviously in a closet. The ventilated closet does not provide total silencing, however, the equipment inside still has to be reasonably quiet.
The Voltage Regulator Module (VRM) is the hottest circuit on the motherboard even with the HSF, at around 70C. Still OK, everything else is much cooler.
If you decide to go with the Ubiquiti US-16-XG 10Gbit SFP+ and 10Gb copper Ethernet switch I know how to silence that sucker as well.
-
@lifespeed The OEM fans arrived... with the mounts, and after some Googling I did get them installed. Serve the Home to the rescue again
I also have the air shroud (MCP-310-50503-0B), which for a mere €15+VAT directs the air over the CPU. I'm not 100% if it's fitted correctly.
On "Optimal settings" in the IPMI, fan speeds are ~4500rpm and the CPU temperatures are at 46C, PCH at 50C and VRMCpu at 54C.
I have two Noctua NF-A4x20 PWM on order - they might be quieter. The current noise is "OK" for my office while I test this setup, and get it bedded down. It's going into a cabinet ultimately in the utility room, so this is probably OK.
(edit)
OK,
screamer"Full" mode is loud at 12800rpm, and gets the CPU down to 34C -
I do have this system now: https://www.supermicro.com/products/system/1u/5018/SYS-5018D-FN8T.cfm
Yeah and the fans are noise at full speed. I use Optimal settings on mine.
I was planing to order the Noctua NF-A4x20, but you need to do some "tweak/hack" on the fan speed control using ipmitool.Using this guide: https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/
But i decide not to do it yeah. :)