Supermicro Xeon D X10SDV-4C+-TP4F with 10Gb SFP+ pfSense compatibility?
-
I would like to build my own 10Gb SFP+ router using a Supermicro X10SDV-4C+-TP4F motherboard. Any suggestions on investigating compatibility of this board, particularly the INPHI CS4227SFP 10Gb SFP+ ports? It also has a pair of 1Gbe I210 LAN ports. This is the 35W TDP Xeon D-1518 4-core.
Right now I have 1Gb/40Mb hybrid fiber coax, but this will likely increase in the future to 2Gb or greater. I need to switch >1Gb on the LAN side to support my 10Gb RAID NAS, future 802.11ax access point at 2.5Gb, and a 10Gb desktop PC. So strictly speaking today I don't have >1Gb WAN to warrant the 10Gb SFP router, but I'm on the brink and don't want to replace my obsolete router with something not spec'd for years of service.
-
@lifespeed See post below.
I think this will come with the 10G 722 NICs that use the ixl driver. There are some issue with VLAN updates when in LAGGs.
https://forum.netgate.com/topic/145869/intel-10g-aq_error-14-did-anyone-find-a-fix-for-2-4-4-p3
We're using SYS-E300-9D-4CN8TP and added additional NIC's to get round this.
Hope this helps.
Hass
-
The motherboard in the server you referenced appears to be X11SDV-4C-TP8F. It has both 10Gb SFP+ as well as 10Gbase-T. The specific SFP+ chipset does not appear to be specified in the motherboard manual. You mention "722 nic's", but it isn't clear these are what is on the X10SDV-4C+-TP4F motherboard referenced in my first post, where the motherboard datasheet diagram shows "INPHI CS4227SFP" for 10Gb SFP+.
Are we talking about the same animal? Also I'm a little unsure of the nature of the problem, if it even applies to my candidate motherboard. Something to do with LAGG (link aggregation)? I read the link you posted, but I'm not quite following exactly what the problem is.
-
@lifespeed I think all the XEON D Supermicro's come with the same 700 10 NIC's.
The ones in ours register as:
dev.ixl.3.%desc:
Intel(R) Ethernet Connection 700 Series PF Driver, VersionThe issue we and others have hard are when you perform Interface/VLAN updates on IXL nic's in a LAGG we get a panic/flapping and we lose the connections for 1-2 minutes. It seems related to using VLAN's
I'm not sure if the issues is there if you're not using a LAG.
It was more a heads up/fyi
Hass
-
I definitely appreciate the heads up on a potentially troublesome problem. Looking at the pfSense bug report it may have been addressed in FreeBSD12?
How would I know if this would affect me or not, I think everybody eventually uses VLAN? I don't plan to use link aggregation, though. In fact my plan is to use one 10Gb SFP+ to the LAN switch, and one 1GB I210 Intel NIC to the WAN.
-
@lifespeed yeah I think it will be fixed in pfSense 2.5 probably some way out. I'm not sure outside of testing it. I know it has the issue when in a LAGG and it's related to VLAN updates but don;t know if using single links if it would be a problem
-
@Hass, I definitely appreciate the heads-up. I'm no IT expert, this is a bit of a stretch for me. But it is clear my 10-year-old "prosumer" router has to go, and I wanted to do something with some growing room.
The error you describe could really stump a novice like myself. If I were to encounter it in my configuration . . .
-
Hi @lifespeed - I'm using the Supermicro 5018D-F8NT system with pfSense, which is based on this board:
https://www.supermicro.com/en/products/motherboard/X10SDV-TP8F
In terms of specifications, this board looks very similar to the one you linked to. In my case, the onboard SFP+ interfaces are actually using the ix (ixgbe) driver in pfSense / FreeBSD as opposed to the ixl drivers.
https://www.freebsd.org/cgi/man.cgi?query=ix&apropos=0&sektion=4&manpath=FreeBSD+11.3-RELEASE+and+Ports&arch=default&format=html
I'm thinking this may be the case for the X10SDV-4C+-TP4F as well. Overall, this setup has been flawless with the exception of this issue (which has only been occurring recently):
https://forum.netgate.com/topic/147301/sfp-ixgbe-network-interface-won-t-come-back-up
Hopefully this will resolved in the next version of pfSense and newer version of FreeBSD.
Hope this helps.
-
@tman222, thanks for sharing your experiences. It sounds a little flaky, but not insurmountable. It sounds like a few people are looking forward to some fixes in the new version, which is not the greatest position to be in.
I'm not sure there are many other good options for 10Gb SFP router hardware.
-
@lifespeed said in Supermicro Xeon D X10SDV-4C+-TP4F with 10Gb SFP+ pfSense compatibility?:
@tman222, thanks for sharing your experiences. It sounds a little flaky, but not insurmountable. It sounds like a few people are looking forward to some fixes in the new version, which is not the greatest position to be in.
I'm not sure there are many other good options for 10Gb SFP router hardware.
Hi @lifespeed - the other option would be to use an SFP+ add-on card. I have used both the Chelsio T520 and T540 SFP+ cards (in fact the T540 is currently installed) and these have been rock solid.
-
Hello
I am looking at this board for a new pfsense server.
Can anyone tell me if all issues were sorted out on pfsense 2.5 for this motherboard?
It really looks like the perfect solution, but I am concerned about the VLAN issues mentioned here. The upgrade to FreeBSD 12 should have fixed it?Thanks
-
@pj I've been using X11SDV-4C-TP8F with 10Gb fiber SFP+ on the LAN side, 1Gb copper Ethernet on the WAN side. I haven't used LAG or VLAN. It works well and has great performance. However, I did use slower, quieter fans on the exhaust, 40mm, 7 total. The passively-cooled CPU really wasn't cutting it, so I found a hard-to-find heatsink compatible with the motherboard that mounts an 80mm fan, cut a hole in the cover, and poked the CPU fan through the top of the 1U rack-mount chassis. Cool and quiet. Really, if you're not willing to put up with screamer fans, this is the only reasonable option.
The performance of the hardware and pFsense has been great. I'm still on 2.4.5, haven't upgraded to 2.5 yet.
-
@lifespeed That is good news.
Those turbine fans are really noisy. I have two older model Supermicro switches that are also too noisy to be used inside of an office, but I have to say that I have not had a single faulty fan on any of my Supermicro servers yet, so I understand why they use turbine fans in everything.
I plan on mounting the server inside of a data center, so it can make as much noise as it wants to.
Thank you for the feedback, and the detailed post.
-
@lifespeed would you be able to share what SFP+ transceiver you are using and a picture of your case mod? It would be very much appreciated.
I'm just building up an X11SDV system in a 1U supermicro case, and the passive cooler does get very warm. I've had one thermal shutdown today before I threw on a bigger fan.
This is a replacement for a pfSense system which I've been running for a few years. Ultimately moving from VDSL to fibre in a few weeks, when I have the new system tuned.
Thanks in advance... Aidan
-
@aidanic I'll see what I can dig up this evening. The heatsink w. fan is a specific Supermicro part, of course.
-
@lifespeed Thanks...
This is my summary so far... like yourself, I'm doing a major bit of future-proofing with this pfSense setup.
I have the X11SDV in a SuperMicro CSE-505-203B 1U case. The case came with few accessories, so I'm waiting on the official 40mm fans (FAN-0065L4). Cable management is painful, even with the M.2 SSD I'm using.
RAM has been a challenge too. After two different sets of RAM failed, I returned the motherboard. It came back two weeks later with "no problems". I then got Samsung RAM which is now working fine - Samsung 16 GB reg. ECC DDR4-3200 DIMM M393A2K43DB3-CWE. I had not expected the motherboard to be quite so picky about RAM modules.
The stock 2.5 pfSense has come up fine with the four 1G ports. When I get the rest of the case assembled, this will be where I start, replacing my existing pfSense setup.
The SFP+ ports are more troublesome. I have a few (random) SFP LC 1G transceivers, and they are returning unqualified sfp+ module detected when plugged in. My single 10G Ethernet transceiver appears OK. I'm waiting on some "Intel" transceivers from FS later this week. While I'm preparing for 10G, all my existing infrastructure is 1G, and my testing opportunities are limited.
Have you done anything with ixgbe settings in loader.conf? Or is it that your transceivers were already on the magic whitelist in the intel firmware?
Many thanks,
Aidan
-
@aidanic I had no difficulty with transceivers, although I did take care to use a model that was compatible with my Ubiquiti US-16-XG switch. Apparently they work fine in the X11SDV as well.
I'm afraid I forgot about the photos and part numbers last night, will try again tonight if I get through taxes.
-
@lifespeed The "Intel" transceivers arrived this morning from FS, and the 10G plug in fine with no error messages. With no other 10G infrastructure, I can't test those end-to-end, although they are probably fine.
The new 1G transceivers also report "unqualified" - and clearly any random mix/match of 10G/1G does not work. I'll go back to look at this after I've sorted out the cooling side of things.
-
@aidanic The part number for the heat sink fan is SNK-C0111AP4L. You'll have to cut a square whole in the 1U lid, I sealed mine with natural rubber electrical tape. Also I recommend using Noctua 40mm fans mounted with silicone nails to quiet it down, but you'll need enough to cover the whole rear panel. Also use a Noctua 60mm(?) heat sink fan, as the OEM Supermicro is a screamer as well. Might be 7 rear fans if I remember correctly. They don't move as much air as the screamers, but it does cool adequately in conjunction with the active HSF.
I actually hacked off the "extra" power supply wires to make room for airflow and wire dressing. I'm not putting any hard drives in this box, so was able to get away with fewer supply wires.
-
@aidanic here's a photo of my whole rack, all I can find at the moment. If you zoom in you can see the HSF sticking out like a supercharger through the hood of an old musclecar. This is before I swapped from the Supermicro HSF screamer fan to Noctua, and removed the tape blocking part of the airflow in the front.
CPU and case fans are all controlled by the motherboard, you can log into the IPMI interface and configure fan control for good cooling with low noise.
