HAProxy with thousand of additional certificates

chaiyut

I have one public ip with about 700 different domains.
I setting in HAProxy frontend and adding additional certificates for every domains.

According to HAProxy document here :

SNI-based multi-hosting with no limit on sites count and focus on
performance. At least one deployment is known for running 50000 domains
with their respective certificates;

That mean HAProxy can handle more than 1000 certificates.

But in Frontend setting gui when I want to edit something I got the message about memory_limit because of loading huge lists of certificate.

I found the configuration to increase memory_limit but I don't think that the right way to fix this problem, that mean in the future I will need to increase again.

Question :

What should I fix this problem.

Informations:

Pfsense 2.4.3-RELEASE
HAProxy 1.7.10-a7dcc3b

kiokoman

the memory_limit is a php settings, not related to haproxy per se, what did you modify? /etc/inc/config.inc ?

// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
ini_set("memory_limit", "512M");
} else {
ini_set("memory_limit", "128M");
}

if the list is that huge i don't think you have other options

pfsense 2.4.3-release is old you should update to 2.4.4-p3 asap, latest haproxy is 1.8.23

kevdog

Current HAproxy release (via HAProxy is 2.1.0)

I have no idea why 2.4.4 ships with 1.8.23. Ridiculous in my opinion.

PiBa

@chaiyut
I'm sure the haproxy binary can handle those amounts of websites/certificates, on a properly configured/tuned machine dedicated to haproxy itself. The webgui in pfSense is a whole other story though.. I think you should probably be investigating to running a dedicated haproxy box and managing it in a totally different manor when dealing with this amount of sites / configuration.. For sure i never intended to make it work for such volumes..

@kevdog
As for the version used.. 1.8.23 is a stable version thats actually stable and the latest in its branche, what you call a current version 2.1 or even 2.0 really hasn't been running 'properly' for that long yet. I do agree that in general the rate of package binary updates is a bit slow.. But anyhow only 1.8 and 2.0 are the currently LTS supported versions of haproxy.. Where the haproxy-2.0 is actually present in the pfSense 2.4.5-RC versions currently available for testing, give it a try? imho the versions used are not 'ridiculous', but yes i guess it could sometimes be nice to get them updated a bit more often.

Anyhow the haproxy-webgui-pfSense-package inside pfSense as present is totally NOT related to haproxy itself and the way they handle it in their loadbalancer-product offerings which probably can handle it with relative ease.. the pfSense webgui package has some totally different and (unknown) restrictions that mostly come from the php/javascript handling.. I for sure have never tried to configure 100+ sites in my haproxy webgui, nor have i tried to push 10Gb/s through a haproxy instance running on pfSense.. even though haproxy itself is probably capable of that as well...

chaiyut

Thank you for your reply.

I'm already increase memory_limit from 512M to 1024M, the page can load properly but slow. I'm thinking about loading only the folder name ( like HAProxy configuration ) and can list all the certificate in another page.

I'm searching and looking in the code for now it not possible.

kevdog

@PiBa

I've heard word of 2.4.5 dropping soon -- I'll probably just wait.

PiBa

@chaiyut
What part of the package is 'to' slow? Is it the 'frontend' overview page? Or is it the configuring of a frontend? If the second then perhaps you can split acls/actions/certificates over more 'shared-frontends' as they are called in the webgui?

Is it possible for you to supply me a pfSense configuration file that you experience the problems with? (just the <haproxy>...</haproxy> would be enough i guess..) Though probably that already contains more info than you can actually share without problems.?. Check carefully for info you want to keep private.. (substituting domainnames and other possibly sensitive info for example1.tld / example2.tld or so..)

Or perhaps just part of the config as you have it for 3 domains? I can probably 'replicate' those 3 to 3000 then to see what your doing..

So I can try and load that part of the config and see if i can maybe work around it or disable part of the webgui buildup in the package if configured.

chaiyut

@PiBa
The part that slow is the second part. I think the idea to add shared-frontend should be the solutions for me. That mean I can modify some configuration except certificates and I don't need to wait for certificates list loaded.

Today story.
I try to add 'shared-frontend' with insensitive domain, it's look like working.
Then I think if I backup configuration and edit xml to move all certificate lists to the new one and restore configuration with the modified xml should be make work easier. So I try with some testing group, but it's not what I thought.
All my site get problem, cannot get the certificate. I decide to restore configuration again with the file before modify.
After reboot I got new trouble about something like:

...
pfsense unable to load dynamic library xmlreader.so
...

I searching and found that this message appear when the upgrade is not complete. ( Ahhhh I don't want to upgrade now !!! ) But I don't have a choice. Checking for command to set ip interface and routing default gateway for upgrade the packages.
After upgrade success, everything back to normal.

Checking memory_limit after upgrade:
Unfortunately memory_limit is back to 512M after upgrade that mean I don't think this is the good way to config, I think maybe should have some place to make custom configuration. ( Not sure maybe not a good idea )

Thank you for your help.

PiBa

@chaiyut said in HAProxy with thousand of additional certificates:

I think maybe should have some place to make custom configuration. ( Not sure maybe not a good idea )

While it sounds like an idea.. there is a issue there with how/who will then write/update the certificate files to the appropriate paths so haproxy can use them.?.
You can put 'advanced' textual configuration in various places in the package though if you wish. All the 3 pages for settings/frontend-edit/backend-edit allow for adding 'advanced' text that will be added as-is to the haproxy config.. In that regard you could copy-paste a haproxy config almost as-is into the global settings and let haproxy use it.. Though probably also not really nice.. I would likely try and use at least some of the frontend webgui options to separate things a bit..

The configuration is here, not sure useful.

Thank you, im seeing that you have all 770 certificates in 1 frontend, tried it here (on 2.5-dev), and same issue "Allowed memory size of 536870912 bytes exhausted"..

Tried a different config with 1000 shared-frontends, each with 1 certificate, and pointing to 1000 different backends a acl and a action for each.. this does 'work' (well haproxy starts with it, and shows stats page havn't tested further..) though some pages aren't exactly loading fast.. (like 10/20 seconds) it doesn't complain about memory that way..

Anyhow ill try and see if i can optimize it somehow to use less memory for the certificate list..

chaiyut

Sorry for weird word. Because of when I searching about memory_limit most of comments is to increase memory_limit configuration, and for pfsense I found to increase is in the file '/etc/inc/config.inc'. But after upgrade this file is override that the reason I said it's not a good idea ( not the right place ) to modify this file configuration.

Thank you, If you want more information or any support from me don't hesitate to ask me.