PHP-FPM webgui crashes and freezes
-
Mmm, that doesn't use PHP directly though. So I assume it slows PHP significantly either by simply using system resources or because something php is trying to do on a large routing table in the gui is causing an issue. For the latter I would expect to see it on some pages only?
-
It will cause all pages to be unable to be opened. The webgui service may even crash.
I read some information and experience and tried it. My point is not to rule out any possibility, but to face the problem and solve it.
What is observed so far is that when the load increases, php-fpm will go to CPU100%. It does not start more php-fpm processes to share the load. It causes the CPU of 1-2 processes to be 100%, so it is easy to cause the web to crash. I also refer to other people's experience sharing to analyze and try. Although I am not a technical developer, when I encounter problems, I don't avoid them, but find the reasons and try to solve them.
https://www.seafog.cn/archives/204286538
https://www.cnblogs.com/boundless-sky/p/9223942.htmlAnother similar situation encountered is that pfsense's frr bgp and zarba each have only one process running, resulting in frequent CPU 100%. However, on Linux, frr, bgp and zarba have 6 processes running respectively. So why does the same frr appear? There is such a difference.
-
this is frr running on linux ubuntu
-
this is pfsense frr, This is what happens during low load conditions
-
pfsense frr
-
The above is the current situation after I modified the php-fpm parameters, and php is basically running normally.
I also searched various resources on the Internet to analyze the results. I am not sure whether it is completely correct. I am just reporting the actual situation to see if it is correct.
-
OK, so the loading is all in zebra rather than in PHP directly. What is the total system loading when it's in that state?
-
But when accessing through a web page, php-fpm often encounters cpu 99.75% or 100%. This is especially obvious when visiting the following two pages:
/index.php /status_frr.php
If you select to display more routes through the page, the pfsense system crashes. When the frr route changes or is sorted, it will cause the pfsense system to crash. When the bgp and zabra process CPU reaches 100%, all web pages will be stuck and unable to work. php-fpm is 100% aslo. This dedicated server only runs pve and pfsense. When frr service is stopped, the web returns to normal.
BGP IPv6 Routes
Display 100 of 11394065 items -
I have tested it on Ubuntu before. When I delete the service integrated-vtysh-config in the vtysh.conf file, frr consumes less resources.
-
Right this feels more like PHP trying to parse some huge route list in a way that was never intended to because it was for IPv4 at the time. So the FRR status page seems likely. But Diag > Routes might also hit something. Or maybe a particular dashboard widget. If you can narrow down exactly which page(s) fail that would be much easier to find the root cause.
-
There are two main aspects that need to be improved. The first frr routing system should not use service integrated-vtysh-config mode.
Should be run with a separate configuration for each function, and use multiple processes. Second, PHP needs to initially use more than 4 processes to reduce the number of requests per process so that other processes can share the load. The code design must adapt to the heavy load of BGP zabra. Currently, I have more than 10 million routing tables for IPv6 alone. If IPv4 is added, it will reach more than 30 million routing tables. Therefore, these situations must be considered, especially when more than tens of millions of routes need to be dynamically optimized for routing and sorting, and the concurrent load will be very large at concentrated times. The above is based on my experience testing on Linux systems. -
So maybe a option to enable a different set of PHP config defaults for systems which are under very high load and have the resources to do so? That could be a feature request.
-
After searching all the information, we found that our PHP setting parameters were incorrect, resulting in the inability to utilize hardware resources.
https://serverfault.com/questions/1055507/php-fpm-using-100-cpu-on-a-server-that-has-16-threads
https://thisinterestsme.com/php-fpm-settings/
Or add a PHP configuration parameter page to allow users to adjust according to their own needs. The most important thing is to adjust the FRR configuration, use multiple processes, and do not use service integrated-vtysh-config, Because in this way, if one function crashes, it will not affect other ones. And in the comparison I made on linux ubuntu, when I changed to not using service integrated-vtysh-config, the resource consumption was significantly reduced, especially the CPU.
Therefore, users can also choose which FRR configuration mode to use.Provide more choices to users because users are more knowledgeable and can adjust according to usage at any time.
-
Both those seem reasonable. I would open feature requests for them.
-
