Possible reasons for why my LAN facing gigabit connection is less than 10Mbit?

[NopIt]

I just can't think of anything that would cause this to happen and I have no clue where to get started on finding the issue.

Let's make this simple:
[Full speed] When doing an Internet speed test on the pfSense box terminal using the speedtest-cli utility.
[Full speed] From my computer to the Internet, when I connect it directly to the Internet (without the pfSense box).
[Very slow speed] When connecting my computer to the pfSense box which is connected to the Internet, I get less than 10Mbit.

My cables:
cat.7 (verified to work at full Gbit speeds)
cat.5e (verified to work at full Gbit speeds)
I also tried about 10 different cables that also work at full speed on other Gbit networks.

My computer:
Gigabit network card (which works at full speed on other networks); No other network cards.
Hardware is also pretty up to date. i7 CPU, 16GB RAM, ... doesn't really matter...

My pfSense box:

pfSense is installed on a 128GB Samsung 850 Evo. (which is the only storage device on that machine)
The peaks in the traffic graphs are the max that I get when doing an Internet speed test on my computer while it's connected to the pfSense box.
CPU and RAM never even gets close to 100%.

[tim.mcmanus]

Why is your WAN IP address 192.168.1.6? What is the device in front of the pfSense box? Cable modem? xDSL modem?

I would have expected your WAN IP address to be an internet-routable address. So I need some additional information to understand your network architecture.

[NopIt]

Thank you for your reply. My WAN IP is 192.168.1.6 because I have to use another router+modem to actually connect with the Internet. I didn't specifically mention that because the issue doesn't seem to be on that side from what I can tell.
(Connecting my computer directly to that non-pfSense router gives me full speed; The pfSense box itself also get's full speed when connected to the non-pfSense router and I run the speedtest-cli utility on the pfSense box.)
The slow speeds only happen on the computer when it has to go through the pfSense box.

[Internet]---[modem]---[non-pfSense router]---[pfSense box] <--- Full speed
[Internet]---[modem]---[non-pfSense router]---[PC] <--- Full speed
[Internet]---[modem]---[non-pfSense router]---[pfSense box]---[PC] <--- Slow speed

[johnpoz]

take internet out of the equation..

PC (iperf server) --- pfsense --- (iperf client) PC

What speed do you see here?

Do a sniff (packet capture) on pfsense lan while doing the test what do you see happening.. Lots of retrans - maybe a problem with the window size not scaling up, etc. etc..

Swap your nic around on the box so your lan nic is now the wan nic - does pfsense still using speedtest-cli get full speed? What is this internet full speed suppose to be?

[heper]

realtek nic on lan perhaps ?

[NopIt]

I'm not home at the moment, but I'll try to set up iperf, package sniffing and nic swapping later today.
Full speed would be around 50Mbit. At the moment I'm not getting above ~5.5Mbit.

Yes, the nics on the pfSense box are both realtek: https://www.gigabyte.com/Motherboard/GA-C1037UN-EU-rev-10#sp

[johnpoz]

There was another thread - I don't recall if he ever came back... But he had put up sniffs because his speed tests where slower than they should of been.. And it was a window sizing issue - you could see from the sniffs that when slow the window size never scaled up..

And when he was seeing normal speed his window size scaled as it should.. I will try and dig up the thread - it wasn't that long ago..

edit: here is that thread couple of months ago - but he never came back
https://forum.netgate.com/topic/136157/sg-1000-throughput-slow-down

So lets also gather some more info.. Your not using transparent proxy right? You using any other sort of packages like snort.

Also what are you rules on your lan are you blocking stuff - like icmp for example?

What about offload settings for tcp segmentation and such in the network advanced section. If I recall the checksum offloading clearly mentions realtek and should prob be disabled. I would prob disable all the offloading and reboot and test your speeds then...

[NopIt]

@johnpoz Thank you so much for all your ideas! My networking knowledge is fairly limited, but I'll try my best.

I set up a Raspberry Pi 3b+ (I tested in advance; when connected directly to the Internet router, it gets the full speed).
Then I set up an iperf3 server and a a dhcp server on it and connected it to the WAN NIC of the pfSense box while the PC was connected to its LAN NIC. Then I ran iperf3 from the PC against the Raspberry Pi and got the same miserable ~5Mbit/s.

I also swapped the WAN/LAN NIC assignment on the pfSense box and did another speedtest from PC through pfSense (and the other router) to the Internet and still got the same ~5Mbit/s (from my PC). And the speedtest-cli utility on the pfSense terminal still got full speed.

I logged an iperf3 run on my PC using wireshark and saved it in a pcap file. It's clean and should only contain the iperf3 related traffic (which is all TCP btw so I don't think ICMP rules would make a difference).
I would appreciate it if you could take a look at it. I'm not sure how to check for the window size scaling:
0_1542839627255_iperf3-log-pcap.zip

I'm not using a transparent proxy or any proxy for that matter.

I don't have any packages installed according to the web interface. I do have speedtest-cli though... but I guess that's a different kind of package?

Here are all my firewall rules:

"Hardware Checksum Offloading" was not disabled. I disabled it now.
"Hardware TCP Segmentation Offloading" and "Hardware Large Receive Offloading" were disabled already.

The Internet speed test from PC through pfSense (and the Internet router) is still at the same ~5Mbit/s.

I'm gonna restart the pfSense box now and see if it helps.

Edit: The restart didn't help.

[Grimson]

Disable your traffic shaping/limiting and test again. If you get better speeds you found your issue.

As for the Realtek NICs, you can try this: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release but if you want full speed and stability you need a different board with Intel or at least Broadcom NICs. As your board only has an old PCI slot and the CPU doesn't support AES-NI I would plan for a complete replacement in the near future.

[NopIt]

F me I feel so stupid right now:

I have no clue why I ever messed around with that...

@Grimson Thank you so much!

I'm really sorry for wasting your time. :(

[johnpoz]

heheheeh - yeah running limiters going to kind of "limit" your speed ;) heheheeh ROFL!!!

Well atleast you found the problem..