New 502 Bad Gateway
-
Wierd, can you reinstall it? 'pkg install -f pfSense-pkg-pfBlockerNG'
Same patch applies without issue on my 2.1.2_1, you havn't replaced anything with one of the previous files from bbcan's links or manually changed any part?Or did you check the 'revert' test results? Below would be the expected test result:
Patch can be applied cleanly (detail) Patch can NOT be reverted cleanly (detail)
Revert only works after the patch is applied already.
-
I think the "Can NOT be reverted cleanly" through me off. I just checked and mine and got the results you show so I applied the patch and it applied successfully. I will report if the 502s go away. Cheers!
-
Same error here. It is quite annoying as my pfsense installation ran smoothly. Since 2.4 I have to schedule reboots.
Any workaround except commenting out the code?
Are the developer working on it? Otherwise I do test the competitor product.Thanks,
-
Fortrash, please try the patch: https://forum.pfsense.org/index.php?topic=137103.msg767259#msg767259
-
Thanks any chance to help the developer?
-
Thanks any chance to help the developer?
How do you mean?
Patches are made by developers.. Testing if it indeed fixes the issue would help, but thats something 'you' (users that experience the actual issue) have to do..
-
Thanks, I am testing the patch.
-
guys check my post here
https://forum.pfsense.org/index.php?topic=110515.msg766964#msg766964
-
Running pfsense 2.4.1-RELEASE with pfBlockerNG on 2.1.2_1 installed and also running OpenVPN server. When the Bad Gateway error happened, OpenVPN clients couldn't connect as well. Rebooting fixed it for now. Any news on when the patch might be rolled into an update? Thanks, Russ
-
PatPend, any news on if the patch helps? ::) may i presume your running with pfBlocker dns blocklists enabled as well?
If there is no positive feedback on its results then there is no need to commit it right? I hope there are no new problems reported by the people that applied the patch, and that they can confirm the problem did not return or at the very least took longer to re-appear.. Would be nice that in a week time they could say it has been running stable..
In the mean time please feel free to apply the patch to your own installation and test it out as well.
-
PatPend, any news on if the patch helps? ::) may i presume your running with pfBlocker dns blocklists enabled as well?
If there is no positive feedback on its results then there is no need to commit it right? I hope there are no new problems reported by the people that applied the patch, and that they can confirm the problem did not return or at the very least took longer to re-appear.. Would be nice that in a week time they could say it has been running stable..
In the mean time please feel free to apply the patch to your own installation and test it out as well.
I applied the patch today, so far so good. I'm running with easylist and easylist privacy plus 28 custom entries to block smart TV traffic. Last time it took about 10 days before hitting the 502 (of course when it did I was out of town when it also took out OpenVPN ::) ).
-
For those willing to give some new code a try i have made a few changes to the 'file locking' code of pfBlockerNG.. :)
Thanks PiBa, hopefully we get some feedback to see if this resolves this issue… and we can get it merged!
-
Hello,
I had to restart my pfsense box again. The patch has already been installed. Should I enable any logging?
Let me know how and what I can provide to resolve the problem.Regards,
-
Hmm ok thanks..
Could you create a file /root/testpfb.sh with following content?
The 'on-mobi.com' is one of the websites mentioned in my /var/db/pfblockerng/dnsblalias/DNSBL_adverts file, change it to something your blocking..:#!/usr/local/bin/php -f error_reporting(E_ERROR | E_PARSE); global $_SERVER; $_SERVER['HTTP_HOST'] = "on-mobi.com"; echo "\nTEST-START\n"; for($i=0;$i<$argv[1];$i++){ if ($i % 100 == 1) echo "."; include('pfblockerng/www/index.php'); } echo "\nTEST-END\n";
Make it executable:
chmod +x /root/testpfb.sh
And create a logfile executing it, this 'should' also hang..:
truss -Haedf -s 100 -o /root/truss_pfblocker_test.log /root/testpfb.sh 1
Preferably create 2 logfiles, 1 while webgui is unresponsive, and one when everything still works. That makes it easier to compare the two.. Also when webgui is unresponsive double check that the culprit still looks like it might be coming from pfBlockerNG..
Below command should return a number above 100../usr/bin/sockstat | grep lighttpd | wc -l
Or actually it might just be waiting for php-fpm which is 'bussy'…
#####################################################
Other separate request, apply the patch below, restart both php-fpm and webgui from the console options 11 and 16.
https://patch-diff.githubusercontent.com/raw/pfsense/pfsense/pull/3769.patch
Then every once in a while run the following on the pfSense box, preferably before everything is already broken or asap after, as it needs a socket available to itself as well it might not work once the problem is presenting itself..:curl -k https://localhost/status
This should show some output on what php-fpm is busy with.. And might show if some/what script is taking more time than it should..
-
Is there an "easy button" for this fix? Maybe just disable pfBlockerNG? I'm pretty uninitiated…just happy I found out "why" it is happening because I had no clue. I'm assuming this will be fixed in 2.4.3 or an update to the extension?
-
I am very surprised that this was not caught in testing: Many, many people run pfBlockerNG, Suricata/Snort and Squid. That should be a basic configuration to be tested.
Yes, it takes traffic and some time to manifest, but any decent QA dept. needs to have, beyond load producing tools, monitoring tools to watch for memory leaks and process status (I did SW QA a few years ago).pfBlockerNG wasn't even part of our QA, we were too busy with 11.0 > 11.1 switch. We love and use pfBlockerNG but it's developed by our community member BBcan177 and not pfSense team (even though we fully support him!). In BBcan177's defense, he had very little time for testing with our switch to FreeBSD version 11.1. To learn about what I am talking about see the following blog post: https://www.netgate.com/blog/no-plan-survives-contact-with-the-internet.html
-
Is there an "easy button" for this fix? Maybe just disable pfBlockerNG? I'm pretty uninitiated…just happy I found out "why" it is happening because I had no clue. I'm assuming this will be fixed in 2.4.3 or an update to the extension?
Disabling pfBlockerNG's DNSBL is a temporary fix until the real cause is found.
-
interestingly following me clean installing 2.4.2 using config.xml (so pfblocker NG updated itself also).
Using my FPM patch I had no 502 errors, and 7 FPM processes were running presumably to satisfy all the php scripts running to process everything and to satisfy the webui requests. So I think my patch works well.
Jim has already gave it the thumbs up on a PR, and Martin has committed the code as well.
https://redmine.pfsense.org/issues/8125#change-35234
-
Hi,
I am currently stable for about 6 days.
2.4.2-RELEASE
pfBlockerNG 2.1.2_1 (updates all lists once a day)
snort 3.2.9.5_3I was just removing the Status_Traffic_Totals (vnstat) package.
Before I was getting 502. -
Hello,
since yesterday I haven't had any problem. No hickups or any breakdown.
The script:
truss -Haedf -s 100 -o /root/truss_pfblocker_test.log /root/testpfb.sh 1
-> does not hang
Question:
What have I to apply to fix the issue? The patch that was made available via git or the fixes below:
https://forum.pfsense.org/index.php?topic=110515.60
Have you already commited the changes?
Thanks,
-
the git patch commit by Martin is the the same as my patch, he just made it an easier process and we can wait for the pfsense staff to approve the commit so it makes it into a future version, but for now either apply my patch in its raw format or the use the git id Martin provided.
As long as your unit has at least a gig of ram it should solve the problem.
Note the patch was created on a 2.4.x unit, I am not sure how clean it will apply on 2.3.x devices, thats untested.
-
Thanks @chrcoluk and @marjohn56,
Hopefully this fixes this issue once and for all…. Looking forward to users feedback!
-
I think the "Can NOT be reverted cleanly" through me off. I just checked and mine and got the results you show so I applied the patch and it applied successfully. I will report if the 502s go away. Cheers!
Reporting back - have not had any issues since the patch was applied. Cheers!!!
-
I had a issue this morning. The only difference was that the update for pfblocker has been enabled again.
Webgui and internal network was reachable.Do you run pfblocker with enabled update? How often do you update?
Thanks,
I think the "Can NOT be reverted cleanly" through me off. I just checked and mine and got the results you show so I applied the patch and it applied successfully. I will report if the 502s go away. Cheers!
Reporting back - have not had any issues since the patch was applied. Cheers!!!
-
… and not a creature was stirring, not even a mouse...
Any additional feedback on the PHP patch posted above would be appreciated! Thanks.
-
… and not a creature was stirring, not even a mouse...
Any additional feedback on the PHP patch posted above would be appreciated! Thanks.
… The stockings were hung by the chimney with care
You only hear when it doesn't work, not when it does.. ;)
-
Hey everyone,
For what it's worth, I found this thread a couple weeks ago, shortly after I installed pfblockerng and configured DNSBL. I had been running my pfsense box for about a year without a single issue, but after setting up pfblockerng and DNSBL, I'd lose GUI and console mgmt access within 24 hours, the only fix being a hard reboot.
I installed PiBa's patch (https://github.com/PiBa-NL/FreeBSD-ports/commit/1766713b26c8f388ad6e7909b2e971f7d74cdfea.patch), and my pfsense box has been running for over 10 days now without a single hiccup. Immediately after installing the patch I noticed that my memory usage dropped from about 50% of my 1GB of RAM to about 30%, and it's remained there ever since.
I did not install chrcoluk's patch, since I didn't want to muddy the waters during my testing. It seems it wasn't needed, at least not in my case. Big thank you to PiBa and all the other folks who helped get this fixed, you guys are awesome! -
Hey everyone,
For what it's worth, I found this thread a couple weeks ago, shortly after I installed pfblockerng and configured DNSBL. I had been running my pfsense box for about a year without a single issue, but after setting up pfblockerng and DNSBL, I'd lose GUI and console mgmt access within 24 hours, the only fix being a hard reboot.
I installed PiBa's patch (https://github.com/PiBa-NL/FreeBSD-ports/commit/1766713b26c8f388ad6e7909b2e971f7d74cdfea.patch), and my pfsense box has been running for over 10 days now without a single hiccup. Immediately after installing the patch I noticed that my memory usage dropped from about 50% of my 1GB of RAM to about 30%, and it's remained there ever since.
I did not install chrcoluk's patch, since I didn't want to muddy the waters during my testing. It seems it wasn't needed, at least not in my case. Big thank you to PiBa and all the other folks who helped get this fixed, you guys are awesome!I can confirm. 12 days without any problems so far!
-
Thanks for the feedback, fyi: pfBlocker 2.1.2_2 includes my patch.
@minterwoot
The reduced memory usage i cant really explain with the changes from my patch.. Maybe it got a bit more efficient but wouldn't expect that to be noticeable by memory usage.. -
I haven't had any issues on any of my boxes for quite a while now on pfsense 2.4.2 and the latest pfblockerng. I was waiting long enough to say with some level of confidence that this issue appears to be resolved but I think it may finally be safe enough to actually say it. Hopefully, that doesn't jinx me!
Much thanks to BBCan117 and everyone else that had a hand in troubleshooting and resolving this. pfblockng is a wonderful tool and I'm happy to be able to use it again without concern.
-
Thanks for the feedback, fyi: pfBlocker 2.1.2_2 includes my patch.
PackageManager shows 2.1.2_1 as the latest available. When will 2.1.2_2 be available ?
I've been running with the quick and dirty fix mentioned earlier in this thread for several weeks now without any 502 happenings . I don't want to go through all the trouble we had at three sites one more time because of the 502 problem .. I must know for sure.
-
For those running 2.4.3, seems it's already running 2.1.2_2
Nice one!
-
Its available on 2.3.5:
https://files00.netgate.com/pfSense_v2_3_5_i386-pfSense_v2_3_5/All/pfSense-pkg-pfBlockerNG-2.1.2_2.txz
https://files00.netgate.com/pfSense_v2_3_5_amd64-pfSense_v2_3_5/All/pfSense-pkg-pfBlockerNG-2.1.2_2.txz
And on 2.4.2:
https://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-pfBlockerNG-2.1.2_2.txz -
I was running 2.4.2 on UFS, and decided to upgrade to ZFS with the config.xml pull.
Prior to the upgrade, I was using maybe 40% memory, and now I am up to 70%.
Still does what pfsense is designed to do.
How can I help out?![Screenshot from 2017-12-22 17-08-58.png](/public/imported_attachments/1/Screenshot from 2017-12-22 17-08-58.png)
![Screenshot from 2017-12-22 17-08-58.png_thumb](/public/imported_attachments/1/Screenshot from 2017-12-22 17-08-58.png_thumb)
![Screenshot from 2017-12-22 17-05-14.png](/public/imported_attachments/1/Screenshot from 2017-12-22 17-05-14.png)
![Screenshot from 2017-12-22 17-05-14.png_thumb](/public/imported_attachments/1/Screenshot from 2017-12-22 17-05-14.png_thumb) -
Not sure if I should open a new thread but my firewall went sideways about an hour ago, running latest released along with PFblocker, ntopng, autoconfigbackup and openvpn. Kids texted me that the internet was down and got home and when I tried to connect received the 502 bad gateway. I am running latest packages for those that are listed.
I haven't pulled the logs completely yet but see this over and over - Could not connect to /var/run/php-fpm.socket.
-
I'm still getting 502 bad gateway error's, they are not often but had my second instance in a month. Attached file contains the debugging steps that has been requested. Restarting PHP-FPM did not resolve the issue, I had to reboot in order to correct my issues. Would like to get this corrected since I've never had the issue in the past.
Let me know if you have any questions.
-
I also still get the Bad Gateway error once a day or so. I will attach the log the next time it occurs.
-
next time it occurs, login to the shell and get a list of the processes running, ideally in verbose format.
-
I am, for the first time, seeing 502 Bad Gateway. Upgraded today to the latest snapshot on my SG-3100.
I do still have internet service so I will wait a bit until someone says they have an idea.Restarted via console, loaded very latest snapshot and now waiting to see if the issue pops up again.
As of 8PM, its an issue again. I guess I need to revert to a previous release and wait a bit for the snapshots.