• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

HAProxy not freeing up inactive memory?

Scheduled Pinned Locked Moved General pfSense Questions
4 Posts 3 Posters 168 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Sherwatt
    last edited by Mar 25, 2025, 11:25 AM

    Hello,

    I suspect that HAProxy might be the cause of sudden memory usage spikes in my pfSense installation.

    I recently experienced pfSense becoming unresponsive and HAProxy's name came up.

    A few minutes ago I checked pfSense and I am seeing things that make me worry.

    On the dashboard I can see this:
    5d1c9b21-217c-4cb8-860a-c3bfbe64e389-image.png
    I am not concerned about memory usage as it seems pretty low.

    However, when I go to Status - Monitoring, this is what I see:
    112fdd8a-db3a-40f4-bcee-6442c66c567c-image.png
    This reports much less free memory.

    I ssh-d into pfSense and ran htop. HAProxy seems to be consuming most of the memory:

    USER      PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED        TIME COMMAND
    www      4433   0.0 39.4 3449360 3187084  -  Is   Mon09       0:28.17 /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -
    

    So, why HAProxy is not freeing up inactive memory? I suspect this might have been the cause of pfSense becoming unresponsive, but I am just guessing.
    I noticed that these spikes probably could be tied to times when someone visits Jellyfin on the local network. A few days ago I was watching a movie on Jellyfin and HAProxy stopped working in the middle of the movie. The service was stopped and in the logs I could see that swapping failed.

    This is the config file for HAProxy. Any idea what could be wrong? Thanks in advance!

    cat /var/etc/haproxy/haproxy.cfg
    # Automaticaly generated, dont edit manually.
    # Generated on: 2025-03-24 09:07
    global
            maxconn                 5000
            stats socket /tmp/haproxy.socket level admin  expose-fd listeners
            uid                     80
            gid                     80
            nbthread                        1
            hard-stop-after         5m
            chroot                          /tmp/haproxy_chroot
            daemon
            server-state-file /tmp/haproxy_server_state
    
    frontend http-to-https
            bind                    192.168.1.1:80 name 192.168.1.1:80
            mode                    http
            log                     global
            option                  http-keep-alive
            option                  forwardfor
            acl https ssl_fc
            http-request set-header         X-Forwarded-Proto http if !https
            http-request set-header         X-Forwarded-Proto https if https
            timeout client          30000
            http-request redirect scheme https
    
    frontend Frontends
            bind                    192.168.1.1:443 name 192.168.1.1:443   ssl crt-list /var/etc/haproxy/Frontends.crt_list
            mode                    http
            log                     global
            option                  http-keep-alive
            option                  forwardfor
            acl https ssl_fc
            http-request set-header         X-Forwarded-Proto http if !https
            http-request set-header         X-Forwarded-Proto https if https
            timeout client          30000
            acl                     torrent var(txn.txnhost) -m str -i torrent.lan.mydomain.com
            acl                     portainer       var(txn.txnhost) -m str -i portainer.lan.mydomain.com
            acl                     pihole  var(txn.txnhost) -m str -i pihole.lan.mydomain.com
            acl                     jellyfin        var(txn.txnhost) -m str -i jellyfin.lan.mydomain.com
            acl                     immich  var(txn.txnhost) -m str -i immich.lan.mydomain.com
            acl                     aclcrt_Frontends        var(txn.txnhost) -m reg -i ^([^\.]*)\.lan\.mydomain\.com(:([0-9]){1,5})?$
            http-request set-var(txn.txnhost) hdr(host)
            use_backend torrent_backend_ipvANY  if  torrent aclcrt_Frontends
            use_backend portainer_backend_ipvANY  if  portainer aclcrt_Frontends
            use_backend pihole_ipvANY  if  pihole aclcrt_Frontends
            use_backend jellyfin_ipvANY  if  jellyfin aclcrt_Frontends
            use_backend immich_ipvANY  if  immich aclcrt_Frontends
    
    backend torrent_backend_ipvANY
            mode                    http
            id                      100
            log                     global
            timeout connect         30000
            timeout server          30000
            retries                 3
            load-server-state-from-file     global
            server                  qBittorrent 192.168.1.120:56656 id 101
    
    backend portainer_backend_ipvANY
            mode                    http
            id                      102
            log                     global
            timeout connect         30000
            timeout server          30000
            retries                 3
            load-server-state-from-file     global
            server                  Portainer 192.168.1.3:9443 id 103 ssl  verify none crt /var/etc/haproxy/server_clientcert_67bb425ba471d.pem
    
    backend pihole_ipvANY
            mode                    http
            id                      104
            log                     global
            timeout connect         30000
            timeout server          30000
            retries                 3
            load-server-state-from-file     global
            server                  pihole_backend 192.168.1.100:80 id 105
    
    backend jellyfin_ipvANY
            mode                    http
            id                      106
            log                     global
            timeout connect         30000
            timeout server          30000
            retries                 3
            load-server-state-from-file     global
            server                  jellyfin_backend 192.168.1.110:8096 id 105
    
    backend immich_ipvANY
            mode                    http
            id                      107
            log                     global
            timeout connect         30000
            timeout server          30000
            retries                 3
            load-server-state-from-file     global
            server                  immich_backend 192.168.1.3:2283 id 105
    
    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by Mar 25, 2025, 5:23 PM

      There's no reason for it to free inactive memory whilst there is still available free memory. It free it eventually. Or it would release it should there ever be any memory pressure from some other service.

      I doubt HAProxy would directly cause a lockup given it never used more than 50% of the RAM. The fact it suddenly spiked like that could be indicative of something else which might though.

      1 Reply Last reply Reply Quote 0
      • G
        giammarcoa3
        last edited by Apr 22, 2025, 8:11 AM

        Hello,
        we experience the same thing. When a file of about 350mb is downloaded haproxy goes from about 50mb to 400mb, continuing to grow until it saturates all ram and restarted by watchdog.
        pfSense 2.7.2, HAProxy version 2.9-dev6-f75a369.
        d3ef2ec7-e1ee-426a-a953-a0cc8711c665-image.png

        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator
          last edited by Apr 22, 2025, 12:25 PM

          HAProxy is restarted? Does it log an error when it stops?

          1 Reply Last reply Reply Quote 0
          • First post
            Last post
          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
            This community forum collects and processes your personal information.
            consent.not_received