Modify Intel X540-T2 NIC To Work At 2.5GbE ?
-
Hmm, hard to see how that would make any difference. But you could try just disabling the monitoring action and not the monitoring itself so you still get quality data.
-
@panzerscope
Try set advertise speed setting
dev.ix.0.advertise_speed=55
NBASE-T Support for Intel X550
You can use system tunables in the GUI now.
Also some explanation why it is not autonegotiating 2.5G https://community.intel.com/t5/Ethernet-Products/Intel-X550-T2-Nothing-above-1Gb-Part-2/m-p/1457923#M32289@Irwan_Intel said in Intel X550-T2 - Nothing above 1Gb - Part 2:
In certain scenarios, where default behavior was to advertise 2.5 Gbps and 5 Gbps, you saw failures to establish the link. But you wont see the failure if both of these speeds are excluded from auto-negotiation.
It was decided to DISABLE these two speeds from auto-negotiation when the driver initializes but still give an end user means to enable them manually.
In fact, i think it's a great success that you managed to connect. Because NBASE-T is a pretty capricious thing and some switches can't activate it if, for example, flow control is disabled.
-
@w0w said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
@panzerscope
Try set advertise speed setting
dev.ix.0.advertise_speed=55
NBASE-T Support for Intel X550
You can use system tunables in the GUI now.
Also some explanation why it is not autonegotiating 2.5G https://community.intel.com/t5/Ethernet-Products/Intel-X550-T2-Nothing-above-1Gb-Part-2/m-p/1457923#M32289@Irwan_Intel said in Intel X550-T2 - Nothing above 1Gb - Part 2:
In certain scenarios, where default behavior was to advertise 2.5 Gbps and 5 Gbps, you saw failures to establish the link. But you wont see the failure if both of these speeds are excluded from auto-negotiation.
It was decided to DISABLE these two speeds from auto-negotiation when the driver initializes but still give an end user means to enable them manually.
In fact, i think it's a great success that you managed to connect. Because NBASE-T is a pretty capricious thing and some switches can't activate it if, for example, flow control is disabled.
Thanks. I have entered in that tuneable and I can now have WAN on Autoselect and it will now negotiate to 2.5G as one would hope, so that is great, thank you.
Oddly however for some reason, my upload is still gimped and I cannot tell just yet if that is the behaviour of the ISP side or Pfsense and the recent change to the X550-T2 NIC and what all that has entailed. I am going to confirm with the ISP that putting their router into Bridge Mode does not effectively half the upload I should be getting (2GB upload plan, though only getting approximately 1GB)
I may have to test after taking their router out of bridge mode to confirm, as even though I have sent in a ticket, I want to validate what they are saying.
EDIT
Looks like there are instances where others are having the same issues, this was a first example.
https://www.reddit.com/r/HomeNetworking/comments/1j95ft2/25_gbit_connection_upload_speed_not_even_close_to/My setup shouldn't be the issue. My networks hierarchy is the following.
ISP Router - In Bridge Mode (2.5G Port)] > Pfsense (X550-T2 - Configured to run at 2.5G) > Zyxel XS1930 L3 Switch (Operating at 10G) > Main PC (Confirmed connected at 10G to the Switch)
-
Are you actually seeing anything over 941Mbps? If not that implies something may still be linked at 1G. Though I don't believe it's possible to link at different rates for up/down so it would have to be at some other part of the connection.
-
@panzerscope said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
ISP Router
Is it Fritz!box? And pfSense is running on bare metal, not in VM?
-
@stephenw10 said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
Are you actually seeing anything over 941Mbps? If not that implies something may still be linked at 1G. Though I don't believe it's possible to link at different rates for up/down so it would have to be at some other part of the connection.
Yes I have seen it spike over 1GB. So I do not think there is a link anywhere in my chain at 1GB limiting me. Though I am going to run my Main PC direct to my Pfsense LAN port to see if I experience anything different, though I am not expecting to.
@w0w said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
@panzerscope said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
ISP Router
Is it Fritz!box? And pfSense is running on bare metal, not in VM?
The ISP Router is an Adtran SDG-8622-20B0 . Pfsense is running bare metal. I would imagine it makes no difference but the ISP router connects to a saparate ONT box, supplied by the ISP. The ONT box is technically the first device in the network but I was informed that the ONT box is capable of 10G.
-
@panzerscope said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
Adtran SDG-8622-20B0
The more I read about similar issues, the more I’m inclined to think that it’s primarily a compatibility problem between the network card’s PHY and the upstream router. Especially if there’s confirmation that everything works as expected when connected directly to upstream router.
And is the presence of the Adtran SDG-8622-20B0 really necessary? If there’s an ONT box with 10G Ethernet, can’t it be connected directly to pfSense? -
How are you testing? What does the pfSense per core CPU usage look like whilst you test?
-
@w0w said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
@panzerscope said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
Adtran SDG-8622-20B0
The more I read about similar issues, the more I’m inclined to think that it’s primarily a compatibility problem between the network card’s PHY and the upstream router. Especially if there’s confirmation that everything works as expected when connected directly to upstream router.
And is the presence of the Adtran SDG-8622-20B0 really necessary? If there’s an ONT box with 10G Ethernet, can’t it be connected directly to pfSense?Well originally when I was testing, I was unable to get an IP when connecting to the ONT, which lead me to believe the ISP router required as some form of intermediary device. However it turns out that it was the initial speed auto negotiation issues I was having with the X550-T2 NIC, rather than an actual issue getting an IP from connecting to the ONT. Now when I connect Pfsense to the ONT, it gets an IP successfully and runs at native 10G.
I am now able to get a much more stable 2G Down and I am now achieving the full 2G Up.
It is now evident that running at 2.5G was either an issue relating to the ISP router that you have eluded to because of PHY reasons, or it is simply down to the NIC drivers not being great of pfsense for 2.5G operation. But for the moment, everything looks to be working fine.
@stephenw10 said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
How are you testing? What does the pfSense per core CPU usage look like whilst you test?
Between 9-10% CPU usage can be observed when running speedtest. I have an Intel Core i5-8500 (6 Core) which fortunately so far has been more than enough for my needs.
Thanks everyone for your help! Ironically I could have just kept with the X540-T2 and connected it to the ONT as it would have just operated at 10G, oh well, to be fair the X550-T2 is the better NIC, so I am not worried.
-
Looks like I spoke to soon.
While I was receiving full 2Gb Upload, it was only temporary. I am now getting around half that again (still connected direct to ONT), sometimes a little lower. Download however is pretty consistent at 2GB. I started to wonder if it was my ethernet cable, but if it is negotiating at 10G and the download is consistent at 2Gb, there would be no reason I can see on my end for the upload to be gimped.
Will have to see what the ISP says. It is either them or some other inconsistency somewhere.
I have also checked the interface statistics and there are no issues reported.
Ironically, just did a speed test again now, I am now getting 2G Upload again. I have to assume for now it is the ISP or some other voodoo.
-
Hmm, it does seem like something outside pfSense then. Urgh.
-
@panzerscope
You can try to swap the ports or play with dev.ix.[port number 0 or 1].fc = 0 tunable or even change the card to X540, but… it looks like some ISP limits. Many of them use the phrase "speeds up to xx Gbps" in the contract, which means that the actual speed can range anywhere from 0 to xx Gbps -
@w0w said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
@panzerscope
You can try to swap the ports or play with dev.ix.[port number 0 or 1].fc = 0 tunable or even change the card to X540, but… it looks like some ISP limits. Many of them use the phrase "speeds up to xx Gbps" in the contract, which means that the actual speed can range anywhere from 0 to xx GbpsThanks. Im leaving everything for now until the ISP come back to me. Oddly, the language used by the ISP is almost contradictory. They state that they guarantee speeds up to the point of their router, but to test at different points of the day to get the best idea of the speeds. In other words they shouldn't used the word "Guarantee" at all as they elude that the speeds can be variable.
Historically speaking, their speeds have been rock solid and extremely small variation has been seen. Their upload has always been consistent, until now where it bounces between full advertised speed and half.
Will report back once I have more info.
-
Hey guys.
While I am waiting on my ISP, I have been doing some research with respects to getting the most out of my X550-T2 NIC.
Regarding the options found in System>Advanced>Networking as highlighted in the below image, should any of these be unchecked?
I also did some other research regarding system tuneable and noted someone with an X550-T2 using the below to enhance performance:
net.inet.tcp.tso="0"
hw.ix.flow_control="0"
hw.ix.num_queues=40
hw.ix.enable_aim=1
hw.ix.max_interrupt_rate=30000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
machdep.hyperthreading_intr_allowed=1I am going to give these a test once I have some feedback, however I have bolded the ones I am unsure of as I believe these are CPU/Hardware specific and wondered if I could get some guidance.
My CPU is shown below.
Could anyone help with some suggested values based on this?
Additionally as per the guide HERE I have added the following.
Happy for other recommendations or any thing I should avoid doing as noted above. While some testing would need to be done, just looking for some "best practice" optimizations concerning my setup.
Many thanks,
P -
When you definitely don't need hn altq support. Those are only found in hyper-V VMs. But it does nothing having it set when you don't have the NICs.
-
@panzerscope said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
hw.ix.flow_control="0"
This one is not working anymore, AFAIK
Use dev.ix.[port number 0 or 1].fc = 0 tunable instead. You can also untick "disable HCO". -
@stephenw10 said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
i226
Thanks for the responses guys. I have made the changes as noted in my prior post. I have an Intel i7 8700 (6 Core/12 Threads) which I am going to go ahead and pop in the firewall as this will give me a higher core clock and hyperthreading.
One question however regarding these variables:
hw.ix.num_queues=6 (Should be set to the number of physical CPU cores you have from what I understand, I set mine to 6 as that is what I have)
machdep.hyperthreading_intr_allowed=1 (Enables the use of logical cores, essentially hyperthreading)The above variables would sort of work in tandem together, so I want to make sure I have them set correctly. Should "hw.ix.num_queues" be set to the value of physical cores ONLY (ignoring logical cores) ? and the "machdep.hyperthreading_intr_allowed" when enabled would make use of any available logical cores.
On another sidenote, I am sourcing a genuine Intel X550-T2. The one I have now is a Lenovo SKU. I am sure the change will make not make any particular difference, but Intel releases its NVM updates more frequently that the sub vendors.
Thanks
P -
@panzerscope
You probably don't need to set those variables. These cards should work at 2 Gbit out of the box. As I already said, try flow control setting.This option might affect something. -
@w0w said in Modify Intel X540-T2 NIC To Work At 2.5GbE ?:
@panzerscope
You probably don't need to set those variables. These cards should work at 2 Gbit out of the box. As I already said, try flow control setting.This option might affect something.Possibly, but I would like to experiment with them all the same :). I did implement the flow control variable, unfortunately there was no improvement, but will leave it enabled anyway as it seems sensible to.
-
Hey all.
So a little update after installing the Intel X550-T2 and updating to the latest NVM (3.7), replacing the Lenovo SKU of the X550-T2 I originally had installed.
I did some further testing for sanity check and to eliminate other devices being the issue.
I hooked up my Main PC using a 10Gbe NIC (X540-T2) to Pfsense using a brand new CAT6 SFTP cable to remove my switch as the middleman and maybe a bad cable. No discernible difference upon multiple speed testing. So I can remove the switch as the issue.
I wondered if it was the cable going from my ONT to Pfsense causing the issue. Again, used a new CAT6 SFTP cable, again no discernible difference.
The only thing I did witness is when swapping out cables, causing a new connection to the ONT, was that when running a speed test immediately afterwards, my upload was at the full 2GB, but would drop off below that after subsequent tests.
Sitting here for the moment, it is either Pfsense/X550-T2 or the ONT that is being difficult.
Worth noting that I am still forcing 10Gbe link speed with the ONT. Auto negotiation ends up with the WAN going up and down, error in the logs show as "arpresolve: can't allocate llinfo for WAN IP HERE on ix0" This is a Kernel level message.
I am now going to disable some of the pfsense services to see if it is that, though I highly doubt it.
