XG-7100 10 gbe throughput issue
-
One of the sfp+ ports on XG-7100 is configured as my LAN interface and is connected to a sfp+ port on my Mikrotik switch. Also connected to an sfp+ port on that switch is a TrueNAS. Using iperf3, data transfers from my TrueNAS to the XG-7100 are happening at 980Mbps (one tenth of what I was expecting). Any ideas of what the problem might be?
-
Sounds like something is negotiating at gigabit, though I would expect to see something around 940Mbit/sec in that case.
During a test, watch
top -aSHrunning on the pfSense node. Report what you see there.
-
980 is pretty freaking high for gig.. I don't think that even possible really.. With Derelict here I would expect something more in the mid 900's if gig was the speed negotiated.. You could get that if talking 1518 frame with the inter-frame gap and preamble.. But if you take those away your more like 974 Max... Guess comes down to how your doing your math.. You could see that with Jumbo I guess.. Taking away all the overhead your prob at 987mbps with jumbo..
-
Here is the iperf3 output. I don't think the connection is being negotiated down to gigabit. Watched top -aSH while running the test. Don't know what I should have been looking for but the iperf3 process was at the top of the list.
[admin@NAS1 ~]$ iperf3 -c 10.10.10.1
Connecting to host 10.10.10.1, port 5201
[ 5] local 10.10.10.10 port 59828 connected to 10.10.10.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 130 MBytes 1.09 Gbits/sec 69 208 KBytes
[ 5] 1.00-2.00 sec 130 MBytes 1.09 Gbits/sec 64 223 KBytes
[ 5] 2.00-3.00 sec 130 MBytes 1.09 Gbits/sec 20 223 KBytes
[ 5] 3.00-4.00 sec 126 MBytes 1.06 Gbits/sec 93 125 KBytes
[ 5] 4.00-5.00 sec 125 MBytes 1.05 Gbits/sec 11 224 KBytes
[ 5] 5.00-6.00 sec 125 MBytes 1.05 Gbits/sec 81 224 KBytes
[ 5] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec 23 224 KBytes
[ 5] 7.00-8.00 sec 126 MBytes 1.06 Gbits/sec 7 224 KBytes
[ 5] 8.00-9.00 sec 97.9 MBytes 821 Mbits/sec 14 224 KBytes
[ 5] 9.00-10.00 sec 123 MBytes 1.03 Gbits/sec 11 224 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec 393 sender
[ 5] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver
-
Your testing too pfsense? Test THRU pfsense..
iperfclient -- pfsense -- iperfserver
-
For testing throughput you should be testing through the firewall, not with one end on the firewall. Not that I think that's exactly why you are seeing what you are seeing, but that methodology is wrong.
-
^exactly... When he said all he saw was iperf using up cpu at the top of his list when you asked him to check top when running iperf I take it that he is testing to pfsense..
-
Let me provide some context. I have 2 machines with 10 Gbe networking capability through sfp+. A TrueNAS Server and the Netgate XG 7100. In about 3 weeks, I will add a virtualization server that also has 10 Gbe sfp+. I am just in the process of setting up the systems for a small business with high throughput needs. The "in rack" networking is at 10 Gbe while for now all the clients are 1 Gbe. What I wanted to test was the throughput within the rack. Since I didn't have another iperf3 server capable of 10 Gbe I used the pfsense box. The other alternative would be to test the throughput to multiple 1 Gbe clients but I don't have that option right now either. Hence the test to the pfsense box. Help me out here but my thought was that if I can't get 10 Gbe (or close) to the pfsense, its unlikely that I will get it through the pfsense to the virtualization server when I set it up in a month's time.
-
Right now we don't know that it's not the switch. Have you tried directly connecting them?
-
Agreed. I am working on ruling that out. What I plan to do is attach the TrueNAS directly to the XG 7100 and run the same test to rule out a switch problem but the pfsense will still be the iperf3 server.
-
I have a couple units here. It'll take me a bit to get the test set up. It's going to be iperf3 between an XG-2758 and XG-7100. If anything is the limiting factor there it should be the 2758.
-
That would be great. Thank you.
-
Today, I attached the TrueNAS directly to the XG 7100 with a DAC sfp+ cable and ran an iperf3 test. The transfer rate was 2.2 Gbps. Twice what is was with the switch between them but still way below what I would have expected. Any suggestions would be greatly appreciated.
-
Clarification on the immediately prior post. On a direct connection of the TrueNAS to the XG 7100, I get a transfer rate of 2.2 Gbps if the XG 7100 is the client. It is ~ 1 Gbps if the XG 7100 is the server. That seems to me to rule out the Switch as the bottleneck. I got exactly the same rates with and without the switch in the path.
-
I was seeing the same sort of thing but it looked like it was the iperf3 process on the firewall itself becoming CPU bound.
That brings us back around to testing through the firewall not to/from the firewall.
-
Thanks for that Derelict. I guess I will just have to wait until the new server arrives and see what happens then. Appreciate your help.
