Performance Tuning for 1.5gbit Internet and 10Gbit LAN

Cryovenom

@sparkyMcpenguin I'm not sure, how would I check that?

@Cryovenom if you're not sure im guessing few, but System > Package Manager

Cryovenom

@sparkyMcpenguin Looks like just one - the openvpn-client-export package. I've got an OpenVPN set up so I can remote back to the house from outside.

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@sparkyMcpenguin Looks like just one - the openvpn-client-export package. I've got an OpenVPN set up so I can remote back to the house from outside.

ok ya i don't really see an issue 'per say' as long as it's not GPON related

maybe filtering ads/malicious ips or things could speed up your network by disallowing some things while allowing legitimate traffic to pass?

you don't have any traffic shaping configured anywhere correct?

Cryovenom

@sparkyMcpenguin no traffic shaping at all. It's basically a bone stock install with my PPPoE for my fiber connection, the OpenVPN, a Hurricane Electric IPv6 tunnel, and a DynDNS client. That's the whole bit.

I don't think blocking ads would help much for speed tests anyway. With one PC attached and no browser open (just the Win 10 Ookla speedtest standalone client) I can't imagine that ads are eating 300mbit off my pipe.

I was wondering if there are any "system tunables" or TCP settings or things like that I could tweak. Or if there are any logs I cpuld look at or tests I could run to try and find out where the bottleneck is.

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@sparkyMcpenguin no traffic shaping at all. It's basically a bone stock install with my PPPoE for my fiber connection, the OpenVPN, a Hurricane Electric IPv6 tunnel, and a DynDNS client. That's the whole bit.

I don't think blocking ads would help much for speed tests anyway. With one PC attached and no browser open (just the Win 10 Ookla speedtest standalone client) I can't imagine that ads are eating 300mbit off my pipe.

I was wondering if there are any "system tunables" or TCP settings or things like that I could tweak. Or if there are any logs I cpuld look at or tests I could run to try and find out where the bottleneck is.

ya im at a loss now, especially once you said pppoe however i will say 15-20% blocking of even just the 935 i have, originally limited it by about 100-200Mbps down to 720-780ish.

maybe it's also it just need to build up the caching database? that's all i got left

Cryovenom

@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.

Do you have any info on how to get started?

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.

Do you have any info on how to get started?

for pfblocker lawrence tech again:
Setup Guide / Tutorial for pfBlockerNG 2.2.5 on pfsense with DNSBL & GeoIP Blocking
i skipped geoip

for snort same yt channel. most of my set up is because i watched this guy pump out videos and watched them over and over before testing the implementation. some things are specific per use case or location client etc. but the majority of what he puts into the videos is a general setup sense (unless video is titled specifically relating to an issue like his codel video.

hope that helps. my IT teacher at school always told me 'Google and Youtube are your friends'... meaning don't trust this junk everest school, trust me the (applying for doctorate) security professional. youtube (from legitimate professionals) has helped me way more than that school ever did (not talking the teacher, they were awesome)

with adding more packages just watch resources. too many lists will eat the RAM just by itself (also there's the disclaimer in pfsense "dont enable all at once')

@sparkyMcpenguin said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@sparkyMcpenguin Well it's worth a shot, plus I've been meaning to set up some DNS ad-blocking at some point anyway.

Do you have any info on how to get started?

pfblocker lawrence tech again:

he also does make some fairly easily understandable explanations as to how certain options or things work (this is why i stayed watching his videos as opposed to people just running through a set up with no explanation)

@Cryovenom i forgot to ask, after settings changes, did you clear (if windows flush) dns through cmd or terminal? that's one of the things the descriptions say you might have to do, after making changes as well - this i did have to do eventually it clears itself though after cron updates but if you want to manually do it that's the way

SteveITS

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

I'm maxing-out around mid-900mbit on download and about 800mbit on upload. The stock ISP modem says it can pull 1200+mbit on the WAN side but only has gigabit ports on the LAN side so it's capped

If you have a 1000 Mbit connection and are getting around 950 I would think that is pretty good given there will be overhead in the packets.

@teamits i agree with this, but OP did say they had their GPON syncing at 2.5Gbps though.. wouldn't that raise the line rate?

Cryovenom

@teamits I effectively have a 1.2gbit internet connection (through a 2.5gbit sync'd GPON) and a 10gbit LAN.

I've got Bell's FTTH service and on their "1Gbit" package most folks over at dslreports day they can pull 1200-1300mbit after overhead (ie: Ookla speed tests showing 1200+mbit) as long as they aren't adding in a 1Gbit bottleneck of their own.

As I mentioned, the ISP's provided equipment can pull as much, but you need at least two devices connected to the modem to take advantage because the LAN-side ports are only gigabit copper. Hence why I removed the ISP router in favour of my pfSense box.

So I should definitely be able to break 1000mbit. I just need a way to find out exactly what my bottleneck is. Am I having too many tx/rx errors? Are my TCP stack settings not optimized for 1+gbit speeds? Is there some kind of hardware limitation?

This is what I'm looking for help with. How to troubleshoot the difference between 900ish and 1200ish. Because I want to upgrade to Bell's 1.5gbit package (on which people are reporting speeds around 1600-1700) but I won't do it until I can prove that my equipment can handle it.

@Cryovenom said in Performance Tuning for 1.5gbit Internet and 10Gbit LAN:

@teamits I effectively have a 1.2gbit internet connection (through a 2.5gbit sync'd GPON) and a 10gbit LAN.

I've got Bell's FTTH service and on their "1Gbit" package most folks over at dslreports day they can pull 1200-1300mbit after overhead (ie: Ookla speed tests showing 1200+mbit) as long as they aren't adding in a 1Gbit bottleneck of their own.

do you know that they are not limiting your speed by the likely same traffic shaping means we have to do the same thing? only curious

also im leaning more toward something on the isp provided equipment now but eh gpon.. pppoe.. no clue on that stuff never had to use set up or diag

is the isp provided equipment forcing dns maybe? overriding your dns? i seem to remember that dns can somehow limit transfer speeds as well... maybe it's more latency related than resource intensity? my latency is 7ms i don't know how fiber latency is other than a google search of others responses.

before i added cloudflare google and quad9 to my dns list, and blocked the isp dns, the dns queries to root servers (and/or intercepted by the ISP) were closer to 20+ms, more during heavy network load (for them, or the neighborhood 'node' - big city).

just remember in regards to pfblockerng dnsbl and snort or suricata they also block other things more than just ads. like malicious hosts spewing junk trying to grab everything they can.

like the speedtest servers themselves load connections that aren't needed to get the speed test functionality (as i noticed during testing) working, and disabling those also (however very slight) increased it a little. snort like to block speedtest servers as well, had to suppress a lot of things, until i dove deeper into openappID and noticed there's rules for many of the snort things that i would constantly unblock. legitimate traffic like youtube (akamai triggers a lot for this). some of them i chose to just turn off where others i left on, and only unblocked it for an internal client that would be the only sender (ie my gaming pc, steam, epic, etc. the other users on this network don't use those services, so by doing it this way, if they installed those services, it would trigger for them and not me)

Cryovenom

@sparkyMcpenguin I appreciate your help and your earnestness, I really do. But you keep drawing conclusions that don't make sense. DNS is the Domain Name System. The only purpose it serves is to translate human-readable FQDNs (fully-qualified domain names) into IP addresses. Once you have the IP address and establish the point-to-point connection DNS has done its job.

So if I was having trouble reaching speedtest.net for example, or one of their servers, or if the resolution of the name into the IP address is slow, then I would possibly have a DNS issue.

As for ISP hardware, there's virtually none of that left in my setup. The GPON module is a small transceiver very similar to the kind that are used for fiber connections in a datacenter, just adapted for long distance bidirectional fiber. Me plugging the GPON straight into my pfsense box would be like if you had a way to plug your Cox cable directly into your pfsense box and remove the DOCSIS modem from the picture entirely.

Like I said, I really do appreciate your eagerness to share your knowledge and help a brother out, but I've been in the industry since probably before your professor who told you to YouTube things. I know DNS, hell I've run DNS servers for hundred-million-dollar-a-year companies. I've worked on large production networks with MPLS connections and I have a 2911 router in the closet just so I can mess around with Cisco IPv6 and VLAN config without screwing things up at work.

What I don't know is enough about the inner workings of pfSense to know how to diagnose my bottleneck. TCP stack tuning for over-1-gbit networks is not something I have much experience in. I don't even have that much FreeBSD experience.

I'm just trying to find someone on here who knows pfsense/FreeBSD well enough to help me troubleshoot this. It's not DNS. It's not likely Adblock. Im losing 300mbit of bandwidth somewhere, be it TCP Window sizing, or TX/rx errors, or driver issues or something. I need someone who knows the nuts and bolts of troubleshooting pfSense to do root cause analysis. So far it feels like we're all still speaking on the "have you tried turning it off and on again" level.

heper

a default install on that hardware should be able to handle up to 2 gbit without any further configuration/tweaking/adjustments.

is this PPoE ? if yes: https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#pppoe-with-multi-queue-nics

if not you could check other possible solutions with regards to broadcom nics:
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#broadcom-bge-4-cards

either way check the entire document to see if anything works

are there any interface errors? (status-->interfaces)

could you provide a 'top' at the time of a speedtest?
it would be ideal to perform a real throughput test with an iperf-server on the wan side , and an iperf-client on the lan side.

Cryovenom

@heper Great! I'll dig into these docs today and see what I find.

As for iperf3 on the WAN side, where would I find a server out on the internet to use as an iperf3 source/destination? Spin up an instance with a cloud provider? Or is there somewhere that offers that as a service?

heper

there are services available online but that would defeat the purpose.
any "old" laptop or pc could be an iperf server

Cryovenom

Any old laptop doesn't have a 10gbit fiber NIC (keep in mind I have no copper ports on my pfSense box at all) and I'd have to remove the ISP's transceiver and change my WAN settings to get rid of the VLAN and PPPoE so I don't know how accurate of a test that would be.

I can set up one of my servers on the WAN side and one on the LAN side each with a 10gbit fiber module, clear out my WAN settings and see what iPerf does. It would at least confirm that the hardware can handle the throughput.