Coming out of hibernation to share my recent pfSense story
-
Good to be back on here! I have moved across the country and have not been as active. Thank you for not deleting my account :)
I'm finally settled in and got my network setup the way I wanted. I have my APU2 with 2.5.2 and a Unifi AP-AC Lite for Wi-Fi. These devices are going through an AT&T fiber gateway BGW210-700 (modem only after some work, another story). All was good and humming along just fine. Then one day, our Google Chromecast with Google TV (CCWGT) started having issues which were mostly noticed when trying to watch VOD or DVR content on YouTube TV. The video would freeze after a few minutes of watching almost like clock work. The connection speed under stats for nerds would drop to a few hundred kilobits, which is not good at all for streaming.Troubleshooting fun begins.
This appeared to be isolated to mainly this CCWGT and mainly noticed when using YouTube TV. We tried other streaming apps and they seemed ok. I normally use streaming apps from my Xbox also, that did not have any issues. Neither did watching live tv on YouTube TV with my phone. Naturally, I suspected the CCWGT was funky. Made sure everything on it was up to date, did resets, factory resets, nothing was helping, surprisingly not even a call to Google support (sarcasm warning). My next step was updating the firmware on my Unifi and making sure the Wi-Fi signal was good. The signal strength was definitely good because the CCWGT is in the same room as the AP and nothing crazy between them. The CCWGT signal reported excellent.Funny how it's always the last thing you check, which brings me to my pfSense story. Maybe I've been using pfSense for so many years that it actually did not benefit me this time around? I have been so used to pfSense being rock solid, that instead of being the first thing I checked, it was the last. I kept saying to myself it can't be pfSense. Even when I would login and take a quick glance at things on my phone, the full dashboard isn't shown unless I scroll down more. I thought maybe this stupid CCWGT needs some more outgoing ports open, so I opened all ports for the CCWGT . Nope, not it. I finally looked at it on my laptop and then it became clear. My pfSense memory usage was at 75% of 4GB and 6% swap. That's when I said oh crap. A pfSense reboot solved all of that including my CCWGT issues. I can't really provide any info on why I had what looked like a memory leak on pfSense. I had done a bunch of things over a bunch of days. Like creating temporary rules and deleting them. Removing packages I no longer used. Enabling and disabling certain rules for my Xbox when needed on a nearly daily basis (thanks GTA online). My rules for the most part are not crazy and I always clean up after myself not leaving behind rules or packages not needed. I barely have any packages running really (cron, backup and ntopng). Had it not been for the CCWGT acting up, I might have not caught it.
Lesson to all, don't overlook pfSense because "it can't be pfSense". This isn't my first time seeing a memory leak which took days to finally start using swap. I have seen this before in our office setup as well. It is very rare, but it is possible.
Oh, and for those trying to troubleshoot a Chromecast with Google TV, don't ping it. The pings to the CCWGT were all over the place. That made me think it had something to do with latency on the Wi-Fi connection or an issue with the CCWGT itself. That was a red herring. I don't think they designed the CCWGT to reply to pings properly. I think as long as you get any response back, that's fair enough. Just don't dwell on that or worry about the odd results.
Happy troubleshooting.
Raffi -
@raffi_ said in Coming out of hibernation to share my recent pfSense story:
memory leak
Probably Memory leak in pcscd.
There is a patch to disable it by default, or you can stop it when you boot your router (stopping it has some issues if you use IPSec so in that case patch and boot).Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
Upvote 👍 helpful posts! -
@steveits Thank you for the tip! I will keep an eye on it.
-
@steveits said in Coming out of hibernation to share my recent pfSense story:
@raffi_ said in Coming out of hibernation to share my recent pfSense story:
memory leak
Probably Memory leak in pcscd.
There is a patch to disable it by default, or you can stop it when you boot your router (stopping it has some issues if you use IPSec so in that case patch and boot).Thanks again, you are the man! Confirmed this was definitely my issue. I saw my inactive memory slowly creeping back up. ps -aux showed that the pscsd was using the most memory. I never even knew about the manual patching package. Cool stuff. I learn something new every time I come back on these forums. So far so good. It looked like there were actually two patches that were part of this same issue according to the history. I wasn't sure if I only needed the second patch or both. The code for both patches were completely different and looked to address different things (pscsd related), so I assumed I did need both and applied both.
-
You should only need this commit to disable PCSCD by default:
afcc0e9c97c1993ae6b95f886665fcb4375d26c7
And that mitigates the issue.Steve
-
@stephenw10 said in Coming out of hibernation to share my recent pfSense story:
You should only need this commit to disable PCSCD by default:
afcc0e9c97c1993ae6b95f886665fcb4375d26c7
And that mitigates the issue.Steve
Thanks Steve, that's good to know. So the first one is the one I really need. I got that applied. I guess no harm in applying the second patch to hide pscsd if it is disabled?
-
Indeed, it won't hurt anything.
