10GB Lan causing strange performance issues, goes away when switched over to 1GB
-
Try running some iperf tests locally across that LAN link. See if you can replicate the same low throughput there.
-
I ran some tests, net result = internal transfer speeds are perfect.
pfSense LAN on 1Gb Port - (Workstation on 1Gb Nic – SFTP File Transfer of 1.5GB ISO file to /Home = 109MiB/s (Full Speed)
pfSense LAN on 10Gb NIC (SPF+ RJ45) - (Workstation on 1Gb Nic – SFTP File Transfer of 1.5GB ISO file to /Home = 110MiB/s (Full Speed)
Internet still suffers when Lan connect to 10Gb, this should be 930Mbps not 190Mbps.
What else can we try, any logs worth pulling or viewing while doing an internet speedtest ?
-
Hmm, you said you tried setting MTU values but this does feel like it could be a fragmentation issue. A packet capture should show that.
Is the speed equally bad in both directions?
-
@stephenw10 I captured a PCAP, nothing is jumping out at me, anything thing specifically I should be filtering for or looking for in regards to fragmentation within Wireshark ?
-
I tried the following wireshark filters
ip.fragment
ip.flags.mf ==1 or ip.frag_offset gt 0
I get 0 returned data, this is leading me to believe there is no fragmentation going on.
-
@stephenw10 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
Is the speed equally bad in both directions?
This could be telling if it's not.
-
@stephenw10 Are you suggesting that I send a large file from the pfsense side to a target SFTP server on my LAN and see if it can sustain the same level of performance as my other tests ?
-
Yes. Or just when you test against fast.com do you also see restricted upload? Assuming your WAN is 1G symmetric.
-
@stephenw10 Ah, sorry, that will not be a good test. I am on cable internet. My download speed is 1Gb but my upload is only 30Mb :( so sadly that test will be of no value.
Anything else we can play with or check in logs, again no fragmentation in the PCAP, looks clean. Its like pfsense is just tanking.
I also tried enabling all the hardware offloading, was previously disabled, no difference.
-
This is interesting.
The port on my switch for the client/workstation shows output drops, this rapidly goes up when I run a speed test.
But the 10GB port uplink to the firewall shows none.
Perhaps the issue is on the Cisco side ?
My Understanding of the 3650 is that it does not have true flow control support
-
To add to this, the Total output drops stop once I switch back to the 1Gb Lan connection.
So there is clearly something happening on the Cisco side regarding the 10GB SPF+ connection in that all the client ports are registering output drops.
-
Hmm, that is curious. You would not think the 10G link should make any difference there. The total rate is still limited by the incoming WAN to less than the 1G link to the client.
But it does start to look like an issue between the switch and client I agree. Try testing from a different client or different NIC type.
I would also try enabling whatever flow control the switch does have. At least as a test.
-
This article seems to describe my issue.
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.htmlSo far I tried disabling QOS on all ports on the switch and the performance has since doubled, getting 600Mpbs now appose to 300Mbps. I am still seeing output drops but not as many, so getting closer. I am at least happy and convinced this issue is purely a Cisco switch issue and not a pfSense bug.
the article is a little confusing but I sill if what they recommend does the trick.
-
Ah, nice. Yeah I would never have suspected that, good catch!
-
@ngr2001 This was discussed 3+ years ago @ this thread
This is a TCP flow control negotiation issue that exists somewhere upstream from the 1GbE LAN client. For me, I am unsure if this is pfSense or the Comcast Cable modem. One way to deal with this is using ethernet flow control but it is an ugly sledgehammer solution.
The Cisco solution is to put this in your 3850 config to increase the buffers for the switch ports that are suffering from output drops:
qos queue-softmax-multiplier 1200
-
Hmm, hard to see how TCP flow control could lead to packet drops from a switch...
Unless the client fills it's buffers and can no longer accept packets maybe...
-
@stephenw10 That is exactly why. TCP flow control negotiation between the source and destination should prevent this from occurring but something is preventing this from occurring. L2 flow control works but is a stupid blunt hammer.
-
Hmm. Well also hard to see how either pfSense or the switch could have any effect on TCP flow between the client and server...
Other than perhaps with the 1G link the flow is sufficiently restricted that the TCP control never comes into effect.
-
@stephenw10 Basic query to ChatGPT or Gemini gives you the similar response. Plus I already dealt with this working at Fortune #1/2 with your appliances.
AI Overview
Learn more
A firewall can potentially disrupt TCP flow control by inspecting and modifying packets in a way that interferes with the mechanisms used to manage data transmission rates between two devices, potentially leading to data loss or congestion issues if not configured properly.
How a firewall could break TCP flow control:
Packet filtering based on TCP flags:
If a firewall aggressively filters packets based on TCP flags like SYN, FIN, or ACK, it could inadvertently drop essential packets used for flow control, like the "window update" packets which signal how much data the receiver can accept.
Deep packet inspection (DPI):
If a firewall performs deep packet inspection on TCP data, it might modify the data stream in a way that alters the TCP sequence numbers, causing confusion in the flow control mechanism.
State-based inspection limitations:
While stateful firewalls track TCP connections, they might not always accurately interpret complex flow control scenarios, potentially causing issues when dealing with large data transfers or dynamic window sizes.
Incorrect configuration:
Misconfigured firewall rules, like overly strict filtering or improper flow control settings, can lead to unintended disruptions in TCP traffic management.
Potential consequences of a firewall disrupting TCP flow control:
Packet loss:
If a firewall drops important flow control packets, the sender might continue sending data faster than the receiver can handle, resulting in data loss.
Congestion:
When flow control is disrupted, network congestion can occur as senders continue to transmit data without receiving proper feedback about the receiver's capacity. -
Mmm, none of that should apply to pfSense unless a user has added complex custom rules or packages.
Potentially there could be some TCP flag sequence that pf doesn't see as legitimate. But I'd imagine that would break a lot of things. We'd be seeing floods of tickets. And pf would log that as blocked packets in the firewall logs (unless that has been disabled).
