Performance Tuning for 1.5gbit Internet and 10Gbit LAN
-
Hi Everyone,
I have a pfSense box that I built out of an HP t620 PLUS not-so-thin-client that I've been using for a little while with an Intel dual-gigabit copper NIC. Recently I upgraded the NIC to a Dell Y40PH (Broadcom 57810S-based dual 10gbit SFP+ NIC) so that I can plug my ISP's GPON module straight into my pfSense box on one port and have 10Gbit fiber to the rest of my network on the LAN side.
The performance is okay, but not reaching what they could/should. I'm maxing-out around mid-900mbit on download and about 800mbit on upload. The stock ISP modem says it can pull 1200+mbit on the WAN side but only has gigabit ports on the LAN side so it's capped. Other folks on my ISP (via DSLReports) say they routinely get 1200+mbit with a similar setup.They helped me get the GPON module syncing at 2.5Gbit. I did the speedtest from a Win 10 gaming PC with another Dell Y40PH. I tested both through my 10gbit switch and also directly fiber patched between the two.
I thought that maybe I was limited by the hardware in the HP t620 (since I was hitting 45% CPU during speed tests) so I decided to try and eliminate that hardware as a variable.
I took one of my two gaming machines (AMD FX-8320 8-core, 16GB RAM), threw a spare hard drive in it and installed 2-4-4-RELEASE-p3 on it, swapping the NIC over. On the LAN side I removed all the rest of my networking gear and plugged a fiber patch straight between that box and my other (hardware identical but with SSD) gaming machine.
On there I got even less, a bit under 800mbit down, 700mbit up. CPU usage during the test was a more respectable 3% (and if I was maxing out a single core it should be closer to 12%) and very little of the 16GB of RAM used so its not CPU or Memory resources.
So I'm thinking that it's something with my config.
Can I get some help troubleshooting and performance tuning this setup?
Cheers,
Jon
-
i too use an 'old gaming pc' as a pfsense appliance heh:
- i have an asym cable gigabit connection (advertised as 935 down 40 up)
I can achieve these speeds without issue*
initially my speeds were about the same as the combo router (would max around 800-850 on my phone - have AC gen2 wifi, those speeds are roughly advertised at about 1700Mbps, so wireless won't have the bandwidth bottleneck)
*= after i added pfblockerng/dnsbl and started tuning snort as well, speeds increased (picture to follow of speedtestnet speedtest current results 10:25PST):
no where near a clue if this actually would or wouldn't help, but for me, roughly 15-20% of the traffic in or out of my residental IP is flagged by DNSBL and pfblocker (mostly just ads and malicious hosts/ips in blocklists) snort also blocks certain traffic but the percentage i mentioned is just what the pfblocker widget shows me - itll say 100% with no network activity other than one session of the dashboard right after a packet counter clear (within 5 minutes a phone calls home and the number drops but eh it's only a representation of the amount done since the last clear)
as for gpon related things im definitely not going to comment on that as i have no clue period hence the cable... also fiber not offered at my address
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
I thought that maybe I was limited by the hardware in the HP t620 (since I was hitting 45% CPU during speed tests) so I decided to try and eliminate that hardware as a variable.
I took one of my two gaming machines (AMD FX-8320 8-core, 16GB RAM), threw a spare hard drive in it and installed 2-4-4-RELEASE-p3 on it, swapping the NIC over. On the LAN side I removed all the rest of my networking gear and plugged a fiber patch straight between that box and my other (hardware identical but with SSD) gaming machine.
On there I got even less, a bit under 800mbit down, 700mbit up. CPU usage during the test was a more respectable 3% (and if I was maxing out a single core it should be closer to 12%) and very little of the 16GB of RAM used so its not CPU or Memory resources.
am running i7 quadcore, 3000 series, with virtualization, 8GB ram. cpu usage usually less than 5% under load maybe 40%ish but that's with multiple device load
ssd install with zfs. i do have swap set up, but pretty sure that wont matter for this thread.
did this on 2.4.4-3 before moving to 2.4.5RC branch. the upgrade didn't affect it
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
I thought that maybe I was limited by the hardware in the HP t620 (since I was hitting 45% CPU during speed tests) so I decided to try and eliminate that hardware as a variable.
out of curiosity how many packages do you have installed/running?
-
@sparkyMcpenguin I'm not sure, how would I check that?
-
@Cryovenom if you're not sure im guessing few, but System > Package Manager
-
@sparkyMcpenguin Looks like just one - the openvpn-client-export package. I've got an OpenVPN set up so I can remote back to the house from outside.
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@sparkyMcpenguin Looks like just one - the openvpn-client-export package. I've got an OpenVPN set up so I can remote back to the house from outside.
ok ya i don't really see an issue 'per say' as long as it's not GPON related
maybe filtering ads/malicious ips or things could speed up your network by disallowing some things while allowing legitimate traffic to pass?
you don't have any traffic shaping configured anywhere correct?
-
@sparkyMcpenguin no traffic shaping at all. It's basically a bone stock install with my PPPoE for my fiber connection, the OpenVPN, a Hurricane Electric IPv6 tunnel, and a DynDNS client. That's the whole bit.
I don't think blocking ads would help much for speed tests anyway. With one PC attached and no browser open (just the Win 10 Ookla speedtest standalone client) I can't imagine that ads are eating 300mbit off my pipe.
I was wondering if there are any "system tunables" or TCP settings or things like that I could tweak. Or if there are any logs I cpuld look at or tests I could run to try and find out where the bottleneck is.
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@sparkyMcpenguin no traffic shaping at all. It's basically a bone stock install with my PPPoE for my fiber connection, the OpenVPN, a Hurricane Electric IPv6 tunnel, and a DynDNS client. That's the whole bit.
I don't think blocking ads would help much for speed tests anyway. With one PC attached and no browser open (just the Win 10 Ookla speedtest standalone client) I can't imagine that ads are eating 300mbit off my pipe.
I was wondering if there are any "system tunables" or TCP settings or things like that I could tweak. Or if there are any logs I cpuld look at or tests I could run to try and find out where the bottleneck is.
ya im at a loss now, especially once you said pppoe however i will say 15-20% blocking of even just the 935 i have, originally limited it by about 100-200Mbps down to 720-780ish.
maybe it's also it just need to build up the caching database? that's all i got left
-
@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.
Do you have any info on how to get started?
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.
Do you have any info on how to get started?
for pfblocker lawrence tech again:
Setup Guide / Tutorial for pfBlockerNG 2.2.5 on pfsense with DNSBL & GeoIP Blocking
i skipped geoipfor snort same yt channel. most of my set up is because i watched this guy pump out videos and watched them over and over before testing the implementation. some things are specific per use case or location client etc. but the majority of what he puts into the videos is a general setup sense (unless video is titled specifically relating to an issue like his codel video.
hope that helps. my IT teacher at school always told me 'Google and Youtube are your friends'... meaning don't trust this junk everest school, trust me the (applying for doctorate) security professional. youtube (from legitimate professionals) has helped me way more than that school ever did (not talking the teacher, they were awesome)
-
with adding more packages just watch resources. too many lists will eat the RAM just by itself (also there's the disclaimer in pfsense "dont enable all at once')
-
@sparkyMcpenguin said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.
Do you have any info on how to get started?
pfblocker lawrence tech again:
he also does make some fairly easily understandable explanations as to how certain options or things work (this is why i stayed watching his videos as opposed to people just running through a set up with no explanation)
-
@Cryovenom i forgot to ask, after settings changes, did you clear (if windows flush) dns through cmd or terminal? that's one of the things the descriptions say you might have to do, after making changes as well - this i did have to do eventually it clears itself though after cron updates but if you want to manually do it that's the way
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
I'm maxing-out around mid-900mbit on download and about 800mbit on upload. The stock ISP modem says it can pull 1200+mbit on the WAN side but only has gigabit ports on the LAN side so it's capped
If you have a 1000 Mbit connection and are getting around 950 I would think that is pretty good given there will be overhead in the packets.
-
@teamits i agree with this, but OP did say they had their GPON syncing at 2.5Gbps though.. wouldn't that raise the line rate?
-
@teamits I effectively have a 1.2gbit internet connection (through a 2.5gbit sync'd GPON) and a 10gbit LAN.
I've got Bell's FTTH service and on their "1Gbit" package most folks over at dslreports day they can pull 1200-1300mbit after overhead (ie: Ookla speed tests showing 1200+mbit) as long as they aren't adding in a 1Gbit bottleneck of their own.
As I mentioned, the ISP's provided equipment can pull as much, but you need at least two devices connected to the modem to take advantage because the LAN-side ports are only gigabit copper. Hence why I removed the ISP router in favour of my pfSense box.
So I should definitely be able to break 1000mbit. I just need a way to find out exactly what my bottleneck is. Am I having too many tx/rx errors? Are my TCP stack settings not optimized for 1+gbit speeds? Is there some kind of hardware limitation?
This is what I'm looking for help with. How to troubleshoot the difference between 900ish and 1200ish. Because I want to upgrade to Bell's 1.5gbit package (on which people are reporting speeds around 1600-1700) but I won't do it until I can prove that my equipment can handle it.
-
@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:
@teamits I effectively have a 1.2gbit internet connection (through a 2.5gbit sync'd GPON) and a 10gbit LAN.
I've got Bell's FTTH service and on their "1Gbit" package most folks over at dslreports day they can pull 1200-1300mbit after overhead (ie: Ookla speed tests showing 1200+mbit) as long as they aren't adding in a 1Gbit bottleneck of their own.
do you know that they are not limiting your speed by the likely same traffic shaping means we have to do the same thing? only curious
also im leaning more toward something on the isp provided equipment now but eh gpon.. pppoe.. no clue on that stuff never had to use set up or diag
is the isp provided equipment forcing dns maybe? overriding your dns? i seem to remember that dns can somehow limit transfer speeds as well... maybe it's more latency related than resource intensity? my latency is 7ms i don't know how fiber latency is other than a google search of others responses.
before i added cloudflare google and quad9 to my dns list, and blocked the isp dns, the dns queries to root servers (and/or intercepted by the ISP) were closer to 20+ms, more during heavy network load (for them, or the neighborhood 'node' - big city).
just remember in regards to pfblockerng dnsbl and snort or suricata they also block other things more than just ads. like malicious hosts spewing junk trying to grab everything they can.
like the speedtest servers themselves load connections that aren't needed to get the speed test functionality (as i noticed during testing) working, and disabling those also (however very slight) increased it a little. snort like to block speedtest servers as well, had to suppress a lot of things, until i dove deeper into openappID and noticed there's rules for many of the snort things that i would constantly unblock. legitimate traffic like youtube (akamai triggers a lot for this). some of them i chose to just turn off where others i left on, and only unblocked it for an internal client that would be the only sender (ie my gaming pc, steam, epic, etc. the other users on this network don't use those services, so by doing it this way, if they installed those services, it would trigger for them and not me)
-
@sparkyMcpenguin I appreciate your help and your earnestness, I really do. But you keep drawing conclusions that don't make sense. DNS is the Domain Name System. The only purpose it serves is to translate human-readable FQDNs (fully-qualified domain names) into IP addresses. Once you have the IP address and establish the point-to-point connection DNS has done its job.
So if I was having trouble reaching speedtest.net for example, or one of their servers, or if the resolution of the name into the IP address is slow, then I would possibly have a DNS issue.
As for ISP hardware, there's virtually none of that left in my setup. The GPON module is a small transceiver very similar to the kind that are used for fiber connections in a datacenter, just adapted for long distance bidirectional fiber. Me plugging the GPON straight into my pfsense box would be like if you had a way to plug your Cox cable directly into your pfsense box and remove the DOCSIS modem from the picture entirely.
Like I said, I really do appreciate your eagerness to share your knowledge and help a brother out, but I've been in the industry since probably before your professor who told you to YouTube things. I know DNS, hell I've run DNS servers for hundred-million-dollar-a-year companies. I've worked on large production networks with MPLS connections and I have a 2911 router in the closet just so I can mess around with Cisco IPv6 and VLAN config without screwing things up at work.
What I don't know is enough about the inner workings of pfSense to know how to diagnose my bottleneck. TCP stack tuning for over-1-gbit networks is not something I have much experience in. I don't even have that much FreeBSD experience.
I'm just trying to find someone on here who knows pfsense/FreeBSD well enough to help me troubleshoot this. It's not DNS. It's not likely Adblock. Im losing 300mbit of bandwidth somewhere, be it TCP Window sizing, or TX/rx errors, or driver issues or something. I need someone who knows the nuts and bolts of troubleshooting pfSense to do root cause analysis. So far it feels like we're all still speaking on the "have you tried turning it off and on again" level.