Slow certificate-related pages
-
Hello,
In our pfSense environment (2 pfSense routers on HA), we have 200 certificates and all the pages where the certificates are listed (oVPN export and Cert Manager for example) are very slow loading, even failing with timeouts (504 on the OpenVPN Export page).
Is this the best way to open an issue ticket? Can we give more information in order to help the issue to be fixed?
Best Regards
-
@aperez said in Slow certificate-related pages:
Can we give more information in order to help the issue to be fixed?
As much as possible ;) Especially details that do not seem seem important.
What type of certs ?I've got about 20 VPN related certificats, some for Freeradius, the OpenVPN server, some Let's encrypt certs, showing them takes not more then a couple of ms.
@aperez said in Slow certificate-related pages:
on HA
You mean : pfSense with a master slave configuration ?
If possible, when you stop that relation, do you still see the same issue ? If so, it could indicate that syncing between the two slows down something. -
We have 200 certificates created on an imported CA, most of them client certificates but some server certificates also. When you click on any link that leads to a page that lists all the certificates it takes forever, no matter which one.
About HA, we have two routers with CARP floating IPs and configuration sync in order to ensure the connection even if one of them goes down, but it does not seem to be relationed with the certificates issue.
Is there any log i can upload for you to check?
-
@aperez said in Slow certificate-related pages:
Is there any log i can upload for you to check?
Not easy to answer on that one.
It would be "anything related to the GUI page you visited that is cert related and happens when you see these slow downs. Non-standard log messages that show up while visiting these pages"(for me) This GUI page : pfSense > Status > System Logs > System > General (all the log pages, depending on what you are using) is more important as the default dashboard page. Because no one cares when everything is ok (aka : the dashboard). The important ones show up on only on the logs pages.
-
@gertjan said in Slow certificate-related pages:
(for me) This GUI page : pfSense > Status > System Logs > System > General (all the log pages, depending on what you are using) is more important as the default dashboard page. Because no one cares when everything is ok (aka : the dashboard). The important ones show up on only on the logs pages.
On this log page I can find this php trace:
2021/01/08 12:20:56 [error] 63930#100510: *435109 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.XXX.YYY, server: , request: "GET /vpn_openvpn_export.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "router01.domain.local", referrer: "https://router01.domain.local/vpn_openvpn_server.php"May i check the php-fpm configuration?
-
What i can see on the processes running when opening this files, is that every time you open this pages it checks all the certificates one by one in order to show the certificate details on the interface, which i think must take some time if you have 200 certs...
-
@aperez said in Slow certificate-related pages:
2021/01/08 12:20:56 [error] 63930#100510: *435109 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.XXX.YYY, server: , request: "GET /vpn_openvpn_export.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "router01.domain.local", referrer: "https://router01.domain.local/vpn_openvpn_server.php"
Exact. That phrase can be shortened to a number. The famous '504' or (PHP) timeout.
@aperez said in Slow certificate-related pages:
What i can see on the processes running when opening this files, is that every time you open this pages it checks all the certificates one by one in order to show the certificate details on the interface, which i think must take some time if you have 200 certs...
That's what I'm thinking.
What I would do, to be sure, is looking at the vpn_openvpn_export.php file. ( /usr/local/www/vpn_openvpn_export.php ).
You will find the function that enumerates the certs.
With the function name, look in the files in /etc/inc/ (grep !) to find where it is defined, and add some log lines to strategic places, something that shows the name of the cert, etc..
Now, visit the page again, and then re check the log file.
You could see now if it is just one cert that takes a long time to get analyzed, or is it "the 200 of them".( I do presume that you have some PHP knowledge, as it is what BASIC was in the eighties, and C in the nineties (and ever there after)).
Again, 20 certs take for me... something like 0,1 sec on a 'low power' bud VM (1 GB memory, one core assigned). If the issue was linear, your would take 2 seconds to open.
Is it possible to remove them all, and add them one by one again, re testing every time.
Or add ten, test, add ten again, and add ? -
After doing some checks on the PHP code, we found the slowliness coming from the /etc/inc/certs.inc file , the "is_cert_revoked" function. It seems that comparing all the certificates with all the revoked certs from the CRL is slow, really slow.
-
@aperez said in Slow certificate-related pages:
the "is_cert_revoked" function
Do you have any revokes certs ?
-
@gertjan said in Slow certificate-related pages:
@aperez said in Slow certificate-related pages:
the "is_cert_revoked" function
Do you have any revokes certs ?
Yes, we have a CRL with nearly 20 revoked certificates. The "is_cert_revoked" function compares all the certificates with all the certificates from the CRL, and this is why it is slow, as having 200 certificates and 20 revoked certificates, makes arround 4000 comparisons, that seem to be slow.
-
@aperez said in Slow certificate-related pages:
200 certificates and 20 revoked certificates
Yeah, that one is known. Admins start to become red if certs have to be revoked. This is just another reason.
Glad you know why now.