1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?

girBot 0

Hi all,

First post here, sorry if its in the wrong section. I did look around.
I came to pfSense from my Asus router as I wanted more networking control.

Just moved office locations.

I confirmed I'm getting 900Mbps direct to my PC wired. When I plug that into pfSense which feeds my 24 port UI switch, all devices cant get more than 40Mbps. Which is massively low compared to my allocated inet speed here. Took me awhile to get all my NAT rules and DNS overrides etc in place and I don't want to tinker too much til I know more about pfSense, so I figured I'd ask for some help here first.

I narrowed the issue down to the pfSense box by doing a speed test directly on it.
The weird part is that my upload from the ISP is 130 and pfSense seems to be reaching that-ish.

There are two PCIe cards in the pfSense server

Intel 10Gbs NIC -> LAN
Intel 1Gbs NIC -> Modem

The onboard NIC is realtek and while it worked briefly it just stopped randomly. It would get a WAN ip but DNS would never work not matter what I did. So just went with the PCIe one

See screenshots below.

Any help would be appreciated.

SteveITS

I regularly see posts here advising people to avoid Realtek due to the FreeBSD drivers.

Have you tried changing patch cables, or setting the LAN port (on pfSense or switch) to Gigabit?

I'm assuming you haven't set up traffic shaping or limiters.

girBot 0

@teamits the switch comes after pfsense, so the 10gbs is auto sensing fine but is also set on the switch.
The 1Gbs to the Modem is direct. I don't see any specifc place in pfSense UI to set the speed of that interface though?

SteveITS

Interfaces->LAN has a "Speed and Duplex" dropdown, at least on the one I looked at. My thought was to set down the 10 Gbit as a test just because that speed is less common? That's just a vague guess though. Long ago I ran into that the other way, where a NIC wouldn't work well at 100 Mbit because (I assume) no one was testing the driver at that speed (in Windows, IIRC).

kiokoman

@girbot-0
did you disable Hardware Checksum Offloading , Hardware TCP Segmentation Offloading, Hardware Large Receive Offloading ? -> System / Advanced / Networking
also there is an official update for realtek driver

girBot 0

@teamits said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

Interfaces->LAN has a "Speed and Duplex" dropdown, at least on the one I looked at. My thought was to set down the 10 Gbit as a test just because that speed is less common? That's just a vague guess though. Long ago I ran into that the other way, where a NIC wouldn't work well at 100 Mbit because (I assume) no one was testing the driver at that speed (in Windows, IIRC).

Yeah I have the speed and duplex in LAN as well. Unfortunately... well fortunately my LAN side is fine.
No speed settings in WAN though.

girBot 0

@kiokoman said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

@girbot-0
did you disable Hardware Checksum Offloading , Hardware TCP Segmentation Offloading, Hardware Large Receive Offloading ? -> System / Advanced / Networking
also there is an official update for realtek driver

Just tried that on your suggestion. Not sure if I need to reboot the router? but i did a couple speed tests. Slightly worse now.

Testing download speed................................................................................
Download: 27.81 Mbit/s
Testing upload speed......................................................................................................
Upload: 83.20 Mbit/s

I'd prefer to use the onboard NIC (the realtek one). The issue I mentioned with that was only last week though. Would I need to install anything separate to "fix" it? The reason I say that is cuz I just did a "lspci" and pfSense isnt even showing the pcie 1gbs nic... which is odd.

pfSense sees it as interface "ue0" but thats not showing up in the devices...

kiokoman

@girbot-0

disable all that stuff, from console:

pkg add https://pkg.FreeBSD.org/FreeBSD:11:amd64/latest/All/realtek-re-kmod-v196.04_2.txz
echo 'if_re_load="YES"' >> /boot/loader.conf.local

reboot

check the boot logs for:

re0: version:1.96.04

also you should do a speed test from a pc on your lan, not from pfsense itself
pfsense is optimized as a router, not as a client

girBot 0

@kiokoman

Not sure if im doing this wrong.. the pkg add command gave a mismatch error due to version. I said yes and it seemed to proceed.

But I don't see anything in the log regarding a version?

kiokoman

@girbot-0
you forgot

echo 'if_re_load="YES"' >> /boot/loader.conf.local

girBot 0

@kiokoman said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

@girbot-0

also you should do a speed test from a pc on your lan, not from pfsense itself
pfsense is optimized as a router, not as a client

I've done it from both.
I just wanted to confirm it wasn't the switch causing the slow down initially, which comes after pfSense.

This is from my workstation which is wired:

girBot 0

@kiokoman said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

@girbot-0
you forgot

echo 'if_re_load="YES"' >> /boot/loader.conf.local

ahh.. well anyway

Ignore the mismatch and continue? [y/N]: y
the most recent version of realtek-re-kmod-v196.04_2 is already installed

so I guess that confirms it.

let me switch WAN over to that interface and see what happens

girBot 0

@kiokoman

Looks like same issue I had last week.

Says its up (its a static WAN ip) but thinks gateway is down. I tried last week to disable gateway monitoring. Same issue though. Its at least recognizing the speed of that realtek card now for WAN.

kiokoman

@girbot-0
i don''t understant
do you have if_re_load="YES" inside /boot/loader.conf.local ?
do you have re0: version:1.96.04 on the boot logs now?
if not, you are not loading the driver at boot

girBot 0

@kiokoman said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

/boot/loader.conf.local

sry sry.. not trying to confuse.

Still nothing in the log mentioning a version :(

girBot 0

@kiokoman

I added this second line as well just to see if it would help since it mentioned to add it during the install of the driver

didnt seem to help though.

kiokoman

@girbot-0
ok it's not loading for me also

KLD if_re.ko: depends on kernel - not available or version mismatch

let me check..

https://drive.google.com/file/d/15lsDnnCz0C9rD9qIsmT0JeniFsSxgxTR/view
this one should load

pkg delete realtek-re-kmod

copy the if_re.ko inside /boot/modules
reboot
and check for
re0: version:1.96.04

girBot 0

@kiokoman

Some success....

but for some reason when using THIS NIC:

Using the other one just works..

kiokoman

@girbot-0
ok, does internet works even if it say gateway offline?
speedtest?

girBot 0

@kiokoman said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

@girbot-0
ok, does internet works even if it say gateway offline?
speedtest?

sorry, shoulda been clearer. no inet. :(
it actually cant reach the GW... see below.

I'll swap the cable back and the interface back to the pcie card, just as a sanity check and change no settings. to see if that comes up still.

girBot 0

@kiokoman

Yah.
Switched back to the other pcie card (which pfsense doesnt seem to know anything about other than ue0). and it works. No changes to config

but speed is crap again.

Testing download speed................................................................................
Download: 32.88 Mbit/s
Testing upload speed......................................................................................................
Upload: 120.32 Mbit/s

I don't understand why switching to the Realtek NIC would cause this.

girBot 0

@kiokoman

Happy to report that it IS now working on the realtek card!

The only change I made was here where it now shows the Speed and Duplex for this card. (it didnt for the other one.). It was set to auto which wasn't working. Setting it to 1000baseT Full made it magically work.

Speedtest results from my machine and the router are same:

Testing download speed................................................................................
Download: 539.01 Mbit/s
Testing upload speed......................................................................................................
Upload: 88.97 Mbit/s

Not the 890 Mbs i get when my machine is wired direct to the modem. But definitely better than 45.
Hopefully tweaking some settings can help boost that a bit more.

Thanks so much for all your help man. Really appreciate the time you spent.

To recap for anyone else that has this issue:

Download the if_re.ko file and put it into /boot/modules on pfSense, a link above is there. Perhaps the pkg add command works as well but didnt for me
Added the following lines to /boot/loader.conf.local (tho the last three I added from another thread. Not sure what they do, will test without)
I did this through SSH, but I think you can also do it in Diag > Edit file from the webgui

if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"
hw.re.msi_disable=1
hw.pci.enable_msix=0
hw.pci.enable_msi=0

Reboot
Webgui: interfaces > WAN re0 > hard set speed to 1000baseT > Save

Started working after that.

girBot 0

@girbot-0

just adding that I set the MTU for the realtek onboard NIC to 1428 and im consitently getting better speed tests.

Testing download speed................................................................................
Download: 707.96 Mbit/s
Testing upload speed......................................................................................................
Upload: 120.81 Mbit/s

JKnott

I have no problem getting well over 500 Mb on my 500/20 connection.

girBot 0

@jknott said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

I have no problem getting well over 500 Mb on my 500/20 connection.

whats your secret?

I'm still getting just about 500 consistently.
If I plug that same cable into my laptop, I get 800-900 :(

JKnott

@girbot-0 said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

whats your secret?

If I told you, I'd have to kill you.

No secret. It just works. I also had 75/10 connecton previously and it regularly exceeded 80 Mb.

girBot 0

@jknott

well I know CAN get 900 sometimes even 1gig. but not on pfsense. and i donno why. dont know if its the realtek adapter or something else.

gabacho4

I'd back up your current configuration, then reinstall pfsense, then run the wizard and create a very basic config. This way you will have standard firewall, NAT, etc rules that are all known to work out of the box. Conduct a test and see what bandwidth you get that way. If that works, then there's an issue with your current config. If not, then it's an issue with your setup (switch, ethernet cables, pfsense box or NICs) because pfsense is certainly able to perform well above what you're seeing. Have you tried to run the test from another device?

SteveITS

@girbot-0 said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

now shows the Speed and Duplex for this card. (it didnt for the other one.). It was set to auto which wasn't working. Setting it to 1000baseT Full made it magically work.

I would guess if the driver doesn't support speed changes it doesn't show. I poked around and on an SG-3100 the LAN doesn't have a speed dropdown...it's a switch so that is meaningless there (the WAN does).

If the port was supposed to autodetect at 1000/full and changing it to 1000/full improved things, I would be looking at the connection...is the patch cable cat 6, etc. IOW that implies autodetect sets to something the hardware can't handle. Autodetect will detect the fastest speed and if the cable is insufficient there will be lots of errors.

girBot 0

@gabacho4 said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

I'd back up your current configuration, then reinstall pfsense, then run the wizard and create a very basic config. This way you will have standard firewall, NAT, etc rules that are all known to work out of the box. Conduct a test and see what bandwidth you get that way. If that works, then there's an issue with your current config. If not, then it's an issue with your setup (switch, ethernet cables, pfsense box or NICs) because pfsense is certainly able to perform well above what you're seeing. Have you tried to run the test from another device?

I'll try it when i can take the inet down. but all my rules are on the LAN side really. So...

girBot 0

@teamits said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

@girbot-0 said in 1Gbps from Modem to PC, capped at 30-40Mbps through pfSense?:

now shows the Speed and Duplex for this card. (it didnt for the other one.). It was set to auto which wasn't working. Setting it to 1000baseT Full made it magically work.

I would guess if the driver doesn't support speed changes it doesn't show. I poked around and on an SG-3100 the LAN doesn't have a speed dropdown...it's a switch so that is meaningless there (the WAN does).

If the port was supposed to autodetect at 1000/full and changing it to 1000/full improved things, I would be looking at the connection...is the patch cable cat 6, etc. IOW that implies autodetect sets to something the hardware can't handle. Autodetect will detect the fastest speed and if the cable is insufficient there will be lots of errors.

Well its weird because it DOES autodetect speed. But internet no work. When I hard set it to what it auto detects it as. It works.
There's nothing in between pfSense and the modem to troubleshoot. It's literally a 6 foot cat7 cable between the two. I tried two cables. Same results. I'm using the same cables for all my wired stuff and everything LAN wise is good.

If I hook up direct PC to modem I get 900+ download speed. pfSense, still around 500. No packet loss or anything. It's like a hard limit somewhere. I'll check the bios, maybe there's an update that might help... I donno.

I'll grab a $35 intel pcie nic off amazon and see if that helps i guess.