pfSense VM latency and WAP performance issues
-
Hi all,
Just joined the pfSense gang 2 days ago with my first pfSense build. Was originally going to install pfSense alone, but was encouraged elsewhere to virtualize and take greater advantage of my hardware. Maybe that was an overreach on my part, but anyway, here's my issue.
I have pfSense 2.4.5 running as a VM in ProxMox 6.1-7. Everything works, it just doesn't seem to be performing as well as I expected. Wired connections are about the same as they were previously latency-wise while my wireless performance has been diminished.
Setup:
I have ProxMox 6.1-7 installed on a Dell 3020 with the above specs. It has a Dell OEM Intel PRO/1000 ET 4-port NIC installed.
The VM router is connected to an unmanaged monoprice 8-port switch that connects to a TP-Link Archer C2 AC750 in AP-mode as well as everything else (wired connections to PC, TV, etc.). Modem is a surfboard 6141 and ISP is Xfinity with 100/6 service.
PVE info:
Latency:
I notice, especially when gaming (rocket league) that my ping spikes pretty dramatically and I see packet losses regularly. This was a problem before migrating to pfSense, but my hope was that with smart queuing that might go away. Maybe this isn't as bad as I think it looks, but it feels pretty bad.
I've read around a bit that CPU and memory spikes are linked to problems like this, but I don't observe that.
Wireless:
Don't really have any data for this, but the wifi connection is much less consistent now even after moving the AP to a central location in my house. DHCP is turned off and connects to the 8-port swithch via a LAN port. Testing with DSLreports gets me speeds of 20-30 mbps and sometimes the test doesn't complete because I lose my connection. WAP has an IP address in my pfSense subnet (192.168.1.27).
Any thoughts? Is this an ISP issue? Do I need to buy a better access point?
Thanks in advance for your help.
-
Ran pingplotter overnight to dig deeper into my problem. Not entirely sure how to interpret the results, but issues seem to crop up between my router and gateway (2nd step).
-
I've got some traffic shaping and float rules set as detailed in this thread.
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815
Have my upload set to 4mbps and down to 100mbps. Speedtest shows without limters I get ~115down/5.5up. Limiters don't Doesn't seem to make a difference, though.
Recently ran Cat6 cable to a few locations. Only heavy usage comes from TV streaming and gaming. Modem is a bit dated, but should be sufficient for my speed.
Wondering if I should go bare metal.
Any help is much appreciated.
-
More tinkering with traffic shaping settings has helped. Also disabled C-state settings in BIOS and ECN for egress limiter, as I read elsewhere that those settings could have a negative impact.
Haven't played with the queue parameters much, though I'm not sure how much impact adjusting those settings will have.
Is there a good way to monitor the change in latency/packet loss in real-time to dial in upload limit? I've seen people post very specific values in that field that I assume they arrived at through testing in that way.Anyway, not sure if the problem is fully solved, or if it was just one good day of results, but glad to see improvement.
-
Spoke too soon. I guess the nice weather meant less demand on the ISP in my area?
What should I try next?
-
Apologies if I've posted in the wrong sub.
Would really appreciate any feedback.
-
There is a specific sub-forum for traffic shaper questions, but I'm not convinced your issue is actually due to a shaping problem. It sounds more like an ISP issue to me.
Check the pfSense system logs for any gateway monitoring issues from
dpinger. Also look at the Gateways tab on the logging screen. Are you seeing any indications of packet loss there? -
Thanks for your reply. Yes, this thread did evolve to include traffic shaping. At the time of my original post 12 days ago I hadn't tried shaping as a solution. As you said, though, I don't think shaping is helping to solve my issue.
Yes, I do see gateway monitoring issues from dpinger. I disabled gateway monitoring actions for both WAN_DHCP and WAN_DHCP6 to keep the gateway from going down. Still get notifications about latency and loss, though.
-
@firerobin said in pfSense VM latency and WAP performance issues:
Thanks for your reply. Yes, this thread did evolve to include traffic shaping. At the time of my original post 12 days ago I hadn't tried shaping as a solution. As you said, though, I don't think shaping is helping to solve my issue.
Yes, I do see gateway monitoring issues from dpinger. I disabled gateway monitoring actions for both WAN_DHCP and WAN_DHCP6 to keep the gateway from going down. Still get notifications about latency and loss, though.
Then I would first concentrate on determing if there is an upstream ISP issue. It looks like it from the information thus far, but there is one other possibility.
pfSense-2.4.5 is based on FreeBSD-11.3-STABLE, and a bug was introduced in FreeBSD-11.3-STABLE that impacts the
pffirewall engine when it manipulates large IP address tables. The impact of that bug is stalling of the network stack. The bug seems more pronounced in virtualized pfSense installs, but can be mitigated - according to some users - by scaling the VM back to only a single CPU core. It takes something that manipulates large IP address tables exceeding 65,535 addresses to trigger the bug. The only things that do that are using the IPv6 bogons table by enabling blocking of bogons on an interface, or by using pfBlockerNG and/or DNSBL with very large IP lists. If the bug impacts you, the latency will be very repeatable and occur every 15 minutes when a cron task executes, or if some other event causes a filter reload command sequence to execute.However, since you mentioned having issues prior to putting pfSense into the mix, I still suspect ISP problems first. And looking at the timing of your
dpingeralerts, I think an ISP problem is more likely. -
@bmeeks thanks again. I'll try lowering the number of cores to 1 and see if that has an impact. The issue I encounter doesn't appear to have a 15 minute periodicity to it. Seems to happen daily when overall usage in my are is high (mid to late afternoon).
Pretty sure I'm not using pfblockerng unless it is installed and active by default. I'll also look whether or not I'm blocking bogon networks.
Assuming it is an isp issue, do I have any recourse beyond choosing another IP? Is there anything I can do on my end to mitigate the problem?
Thanks again for your help.
-
@firerobin said in pfSense VM latency and WAP performance issues:
@bmeeks thanks again. I'll try lowering the number of cores to 1 and see if that has an impact. The issue I encounter doesn't appear to have a 15 minute periodicity to it. Seems to happen daily when overall usage in my are is high (mid to late afternoon).
Pretty sure I'm not using pfblockerng unless it is installed and active by default. I'll also look whether or not I'm blocking bogon networks.
Assuming it is an isp issue, do I have any recourse beyond choosing another IP? Is there anything I can do on my end to mitigate the problem?
Thanks again for your help.
pfBlockerNG is not installed by default. In fact, no packages are installed by default (I'm talking about the ones available on the SYSTEM > PACKAGE MANAGER tab).
Whether or not you can do something on your end for the ISP issue depends on exactly what the problem is. Cable system Internet technology uses a device called a CMTS (Cable Modem Termination System). The CMTS is generally connected via optical fiber to various nodes around the system. These nodes in turn translate the signal into something suitable for transmission over coaxial cable. The nodes are the distribution points. If the node serving you is saturated or fully subscribed, performance can be lacking during peak traffic times. Nothing you can do about that but maybe complain enough to get the cable operator to increase the node capacity.
The other possibility is you have a marginal signal at your cable modem. You can call up the stats page in your modem and see what the SNR (signal-to-noise ratio) readings are for each carrier being used to send you signal. You can also see the transmit power your modem is having to use to reach the CMTS node serving you. Google "typical cable modem stats" to see what normal values usually look like. If you have a marginal signal, you can ask the cable company to come investigate. Maybe the coax drop from the street into your home is flaky. With cable modem signals, outside temperature has an influence. Very hot temps can lower the SNR, and if you were marginal with SNR to begin with, that can put you down in an area where the signal reliability is poor. If the case is just that you have a long drop with a lot of loss, sometimes a bi-directional amp placed out at the pole where your drop originates can help. For my setup, my house is approximately 350 feet from the street where my cable modem signal originates. Even with good cable, that length of coax has significant loss at 500 Mhz and above where cable modem download signals live. My service is underground from the pole at the street into my home. So at the pole, where the signal coax comes down and enters the ground, I inserted my own outdoor rated bi-directional amp there at an existing splice (F56 coupler, actually). The amp is powered over the coax by a power inserter in my house. That amp boosts both the download signal coming from the CMTS node and my upload signal returning to the CMTS node. It helped me greatly. Prior to that I was getting signal drops during the hottest parts of the day in summer (it gets really hot where I live -- as in 95 degrees and up in summer). Since installing the amp, I've had zero issues and routinely have 250 day or longer up times. Note that an amp will only help if you truly have low SNR due to high drop cable losses. And the amp works best when it is placed at the street. It does not help much when located in your home because by then the signal loss in the coax drop has already happened and the SNR is degraded. All an amp will do at that point is just amplify the noise and actually make the SNR worse.
One last thing I will mention. A 100/6 line is fairly asymmetrical. If someone is downloading fairly heavily, that will make it easy to saturate the upload. And once that happens the connection as a whole goes to "you know where" in a handbasket as we say in the south ...
. That is a point where upload traffic shaping can help. But first you would want to positively identify that as the issue. You would know upload saturation is a potential culprit if the latency only pops up when your download is heavily utilized. -
Modem on this list? https://badmodems.com/Forum/app.php/badmodems
-
SB6141. Doesn't look like it's on the list, unless it has a second designation.
I've had it a few years now, but figured it wasn't a necessary upgrade given my speed.
-
Thanks again for all the information. I'll look in the areas you've pointed out to see what I can find.
I have PPC evo1-9U/U amp in the basement that I bypassed. I assumed it was for splitting cable signal to multiple rooms and since I'm only running the one cable to my modem, I thought it wasn't needed. Would it be worth plugging back in to it?
-
What model modem do you have and what do its signal levels look like (if you can see them.. ) http://192.168.100.1
-
@firerobin said in pfSense VM latency and WAP performance issues:
Thanks again for all the information. I'll look in the areas you've pointed out to see what I can find.
I have PPC evo1-9U/U amp in the basement that I bypassed. I assumed it was for splitting cable signal to multiple rooms and since I'm only running the one cable to my modem, I thought it wasn't needed. Would it be worth plugging back in to it?
Usually an amp placed inside your home, unless the drop from the street to your house is extremely short, does not do as much for you as an amp placed at the drop on the street. The amplifier is to make up for cable losses (as in loss of signal as it travels through the coax from the nearest cable system amplifier to your modem).
Now, if you do indeed have any splitters that are in front of your modem, then you might well benefit from putting them AFTER your modem if possible. This is especially true for passive splitters. Any amplifier you use must have the proper bandwidth. That usually means an amp that is good from 20 - 1000 MHz is needed.
However, don't get too distracted by this business about amps until you check your stats and see what the signal levels and SNR values are. Here is a screenshot of my RF stats --
I have 100/10 service from my provider. My modem is an ARRIS DOCSIS 3.0 / SIP 2.0 Touchstone Residential Gateway, Model TG2472G. It is provided by my cable company, and I used their modem since I have residential phone service through them as well.
The amplifier you mentioned you have is fine for multiple drops inside your home, but it has unity gain for the return path. That means it only makes up for the loss caused by the splitter inside the amp. It does not have gain for making up the signal losses you might experience coming into your home. The amplifer I have (forgot the model number and the long exposure to the elements on the pole have faded the sticker on the box) has adjustable upstream and downstream gain. That allows me to boost both the downstream (from the CMTS to my modem) and the upstream (return path from my modem to the CMTS) signal levels. I adjusted them until I had power levels showing in the modem stats that were close to optimal.
-
My modem is an Arris Surfboard SB6141. The P/N is 581901-022-00 if that is useful.
Here's the readout for signal levels.
-
@bmeeks No splits in my coax. I do have a union connector since the original drop from the pole inside terminated at the amp in the basement. I had to connect another cable to that with the union to make it long enough to reach my first floor closet.
-
Those signals are fine. That is an older modem. You could be suffering from congestion on the channels you are bonded on.
-
I agree with @chpalmer. Your signal levels and downstream SNR look okay. It could be congestion on your local node, or perhaps there is a physical issue with that node itself. Usually, you would hope the cable company (especially a large one) has some central monitoring infrastructure that would spot issues and alert someone. For instance, a bad fiber card or something else.
Do you have any tech-savvy neighbors that you could compare Internet performance and reliability with (that are using the same ISP as you are)? If they are having similar issues, then it would definitely be on the ISP side and maybe enough complaining (the old squeaky wheel thing) would get some help.
It can be really frustrating to get some types of problems taken care of because you have to wade through the first-line call center support. They work totally off of scripts and standard troubleshooting charts. Anything not in the script blows their mind and they usually try to hurry you off the line...
.When I built the house I am in now and moved in, the only decent option was 6 Meg ADSL. It worked fine for the time (back in 2003 and 2004). But I suddenly started having weird speed issues with related download problems. Tried first line support without much success. Fortunately, a much longer time ago in the 90s, in my older home, I had subscribed to some of the first DSL circuits in town. Due to some initial rollout bugs, I had the privilege of working with one of the phone company engineers. I still had his name and email address in 2004, so when I started having the problem I got in touch and he routed me to the right engineers to discuss the issue. After some back and forth over the phone they determined they had an issue with the fractional DS3 feeding the remote DSLAM. They were using I think 8 bonded DS1 links to feed the remote DSLAM from the Central Office using the equivalent of LACP. That bonding board was flaky and kept dropping out some of the DS1 feeds. They had the local tech replace that board and problem solved.
