Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Unholy device eats 100% cpu php-fpm by accessing captive portal like hell

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    15 Posts 4 Posters 4.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      deajan
      last edited by

      Hello,

      [EDIT] This is not an upgrade issue as it seemed[/EDIT]

      I have a couple of SG 2440 devices runnnig with mostly identical configuration (clones from a master).

      I've updated two of them from pfSense 2.3.1-1 to 2.3.1-5.
      One of the two devices now shows permanent 100% cpu usage since the upgrade (was 30-40% before).
      If I happen to reboot it, cpu goes 100% after a couple of minutes again.

      Here's what I get from ps aux

      
      USER      PID %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
      root    83276 31.3  0.9 274440  37832  -  R     6:16PM  0:06.94 php-fpm: pool nginx (php-fpm)
      root    64146 31.0  0.9 274440  37728  -  R     6:16PM  0:10.65 php-fpm: pool nginx (php-fpm)
      root    99202 23.5  0.9 274440  37836  -  R     6:17PM  0:03.62 php-fpm: pool nginx (php-fpm)
      root       12 22.0  0.0      0    416  -  WL    3:28PM 20:26.92 [intr]
      squid   61240 20.9 15.4 737832 640356  -  S     3:29PM 12:00.18 (squid-1) -f /usr/local/etc/squid/squid.conf (squid)
      root       11  3.9  0.0      0     32  -  RL    3:28PM 95:49.77 [idle]
      root    50094  0.8  0.2  38848   7600  -  S     3:28PM  0:33.70 nginx: worker process (nginx)
      root    49329  0.5  0.2  38848   7560  -  S     3:28PM  0:34.55 nginx: worker process (nginx)
      squid   87161  0.4  0.3  33564  12040  -  S     4:57PM  0:13.72 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      root    38758  0.1  0.1  14508   2320  -  Ss    3:29PM  0:54.43 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf -b 127.0.0.1
      root        0  0.0  0.0      0    304  -  DLs   3:28PM  1:54.39 [kernel]
      root        1  0.0  0.0   9136    820  -  ILs   3:28PM  0:00.03 /sbin/init --
      root        2  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [crypto]
      root        3  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [crypto returns]
      root        4  0.0  0.0      0     32  -  DL    3:28PM  0:00.03 [cam]
      root        5  0.0  0.0      0     16  -  DL    3:28PM  0:08.31 [pf purge]
      root        6  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [sctp_iterator]
      root        7  0.0  0.0      0     32  -  DL    3:28PM  0:01.03 [pagedaemon]
      root        8  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [vmdaemon]
      root        9  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [pagezero]
      root       10  0.0  0.0      0     16  -  DL    3:28PM  0:00.00 [audit]
      root       13  0.0  0.0      0     32  -  DL    3:28PM  0:00.00 [ng_queue]
      root       14  0.0  0.0      0     48  -  DL    3:28PM  0:00.46 [geom]
      root       15  0.0  0.0      0     16  -  DL    3:28PM  0:19.30 [rand_harvestq]
      root       16  0.0  0.0      0     80  -  DL    3:28PM  0:00.54 [usb]
      root       17  0.0  0.0      0     16  -  DL    3:28PM  0:00.02 [idlepoll]
      root       18  0.0  0.0      0     32  -  DL    3:28PM  0:01.22 [bufdaemon]
      root       19  0.0  0.0      0     16  -  DL    3:28PM  0:03.63 [syncer]
      root       20  0.0  0.0      0     16  -  DL    3:28PM  0:00.06 [vnlru]
      root       56  0.0  0.0      0     16  -  DL    3:28PM  0:00.35 [md0]
      root      274  0.0  0.6 270344  24804  -  Ss    3:28PM  0:04.39 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
      root      313  0.0  0.1  18888   2444  -  INs   3:28PM  0:00.03 /usr/local/sbin/check_reload_status
      root      315  0.0  0.1  18888   2288  -  IN    3:28PM  0:00.00 check_reload_status: Monitoring daemon of check_reload_status
      root      327  0.0  0.1  13624   4836  -  Ss    3:28PM  0:00.01 /sbin/devd -q
      root     3066  0.0  0.0  12268   1872  -  Is    3:29PM  0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
      root     3306  0.0  0.0  12268   1884  -  I     3:29PM  0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
      root     3646  0.0  0.0  12268   1872  -  Is    3:29PM  0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
      root     3819  0.0  0.0  12268   1884  -  I     3:29PM  0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts  (minicron)
      root     3961  0.0  0.0  12268   1872  -  Is    3:29PM  0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
      root     4510  0.0  0.0  12268   1884  -  I     3:29PM  0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data  (minicron)
      root     6586  0.0  0.2  59068   6348  -  Is    3:28PM  0:00.00 /usr/sbin/sshd
      root     6791  0.0  0.1  14612   2108  -  Is    3:28PM  0:00.01 /usr/local/sbin/sshlockout_pf 15
      root     7335  0.0  0.1  21624   5696  -  Ss    4:57PM  0:00.85 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
      nobody   9957  0.0  0.1  16836   4824  -  Ss    4:57PM  0:15.62 /usr/local/sbin/darkstat -i igb0 -b 172.16.1.1 -b 127.0.0.1 -p 666
      nobody   9971  0.0  0.1  16836   2368  -  Is    4:57PM  0:00.00 darkstat: DNS child (darkstat)
      root    10295  0.0  0.1  16676   2232  -  Ss    3:28PM  0:05.41 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
      squid   10929  0.0  0.1  37752   4084  -  S     3:46PM  0:01.62 (pinger) (pinger)
      root    11185  0.0  0.1  18896   2400  -  Is    3:28PM  0:00.02 /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid
      root    12579  0.0  0.4 776064  18252  -  Is    4:57PM  0:00.04 /usr/local/sbin/radiusd
      root    13546  0.0  0.1  16652   2104  -  S     6:17PM  0:00.00 ping -c 2 -i .2 172.16.1.62
      root    19086  0.0  0.2  42836   6864  -  I     4:57PM  0:00.00 /usr/local/sbin/syslog-ng -p /var/run/syslog-ng.pid
      root    19305  0.0  0.2  58280   8860  -  Ss    4:57PM  0:02.86 /usr/local/sbin/syslog-ng -p /var/run/syslog-ng.pid
      squid   21185  0.0  0.1  37752   4084  -  S     3:35PM  0:01.61 (pinger) (pinger)
      squid   22991  0.0  0.1  37752   4084  -  S     3:46PM  0:01.84 (pinger) (pinger)
      root    23250  0.0  0.1  14428   2116  -  I     3:29PM  0:00.01 /usr/libexec/getty al.Pc ttyv0
      root    23348  0.0  0.1  15012   2292  -  Ss    4:57PM  0:00.91 /usr/local/bin/dpinger -S -r 0 -i CompletelFibre -B XX.XX.XX.XX -p /var/run/dpinger_CompletelFibre_XX.XX.XX.XX_8.8.4.4.pi
      root    23384  0.0  0.1  15012   2292  -  Ss    4:57PM  0:00.90 /usr/local/bin/dpinger -S -r 0 -i CompletelFibre20 -B YY.YY.YY.YY -p /var/run/dpinger_CompletelFibre20_YY.YY.YY.YY_208.67.2
      root    23422  0.0  0.1  14612   2180  -  Is    3:29PM  0:00.00 /usr/local/sbin/sshlockout_pf 15
      unbound 26075  0.0  0.8  59220  32612  -  Ss    4:57PM  0:25.63 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
      root    26514  0.0  0.1  38848   5876  -  Is    3:28PM  0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
      root    26813  0.0  0.2  38848   7000  -  S     3:28PM  0:02.10 nginx: worker process (nginx)
      root    26857  0.0  0.2  38848   7036  -  S     3:28PM  0:02.23 nginx: worker process (nginx)
      root    26989  0.0  0.1  16532   2260  -  Ss    3:28PM  0:00.13 /usr/sbin/cron -s
      root    30368  0.0  0.4  30140  17968  -  Ss    3:28PM  0:00.66 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
      dhcpd   30747  0.0  0.4  24800  15036  -  Ss    4:57PM  0:01.30 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igb0
      root    31063  0.0  0.2  82268   7372  -  Ss    6:14PM  0:00.06 sshd: root@pts/0 (sshd)
      root    31868  0.0  0.1  17000   2584  -  IN    4:57PM  0:01.42 /bin/sh /var/db/rrd/updaterrd.sh
      squid   32799  0.0  0.1  37752   4084  -  S     3:35PM  0:01.66 (pinger) (pinger)
      root    41937  0.0  0.1  16532   2264  -  I     6:15PM  0:00.00 cron: running job (cron)
      root    42288  0.0  0.1  17000   2496  -  Is    6:15PM  0:00.01 sh /usr/local/bin/rangecheck-launcher.sh
      root    43020  0.0  0.1  17760   3224  -  S     6:15PM  0:00.22 bash /usr/local/bin/rangecheck.sh
      root    48914  0.0  0.0   8168   1824  -  IN    6:16PM  0:00.00 sleep 60
      root    49230  0.0  0.1  38848   6228  -  Is    3:28PM  0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-lan-CaptivePortal.conf (nginx)
      root    49536  0.0  0.2  38848   7484  -  S     3:28PM  0:32.72 nginx: worker process (nginx)
      root    49784  0.0  0.2  38848   7572  -  S     3:28PM  0:36.73 nginx: worker process (nginx)
      root    49852  0.0  0.2  38848   7532  -  S     3:28PM  0:30.47 nginx: worker process (nginx)
      root    50402  0.0  0.2  38848   7540  -  S     3:28PM  0:37.58 nginx: worker process (nginx)
      root    50506  0.0  0.0  12268   1876  -  Is    3:28PM  0:00.00 /usr/local/bin/minicron 60 /var/run/cp_prunedb_lan.pid /etc/rc.prunecaptiveportal lan
      root    50763  0.0  0.0  12268   1884  -  I     3:28PM  0:00.00 minicron: helper /etc/rc.prunecaptiveportal lan (minicron)
      squid   52008  0.0  0.3  33564  11212  -  I     5:29PM  0:00.12 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   52078  0.0  0.3  33564  11132  -  I     5:29PM  0:00.12 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   52268  0.0  0.3  33564  11048  -  I     5:29PM  0:00.09 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   52656  0.0  0.3  33564  10932  -  I     5:29PM  0:00.12 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      root    60835  0.0  0.3  55860  13028  -  Is    3:29PM  0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
      mysql   62083  0.0  0.1  17000   2484  -  Is    3:29PM  0:00.03 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file
      squid   65001  0.0  0.1  37752   4084  -  S     4:57PM  0:00.96 (pinger) (pinger)
      squid   66407  0.0  0.1  37752   4080  -  S     3:29PM  0:01.63 (pinger) (pinger)
      root    67386  0.0  0.0   8168   1824  -  I     6:16PM  0:00.00 sleep 55
      squid   68772  0.0  0.3  33564  11596  -  S     5:19PM  0:00.09 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   68796  0.0  0.3  33564  11476  -  I     5:19PM  0:00.12 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   68989  0.0  0.3  33564  11252  -  I     5:19PM  0:00.08 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   69238  0.0  0.3  33564  11252  -  I     5:19PM  0:00.08 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      root    76722  0.0  0.9 274340  38936  -  S     6:09PM  0:00.81 /usr/local/bin/php-cgi -f /etc/rc.prunecaptiveportal lan
      squid   78216  0.0  0.1  37752   4084  -  S     3:29PM  0:01.64 (pinger) (pinger)
      mysql   84761  0.0 11.3 683080 469892  -  I     3:29PM  0:03.21 /usr/local/libexec/mysqld --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-di
      root    87431  0.0  0.0  14408   1952  -  Ss    3:29PM  0:01.31 /usr/sbin/powerd -b hadp -a hadp -n hadp
      squid   87539  0.0  0.3  33564  12032  -  S     4:57PM  0:01.82 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   87589  0.0  0.3  33564  12032  -  S     4:57PM  0:00.56 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   87655  0.0  0.3  33564  12032  -  S     4:57PM  0:00.31 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   87901  0.0  0.3  33564  12016  -  S     4:57PM  0:00.17 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   87914  0.0  0.3  33564  12016  -  S     4:57PM  0:00.16 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   88067  0.0  0.3  33564  12016  -  S     4:57PM  0:00.10 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   88124  0.0  0.3  33564  11796  -  S     4:57PM  0:00.12 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
      squid   88237  0.0  0.1  37752   4084  -  S     4:57PM  0:01.34 (pinger) (pinger)
      root    91247  0.0  0.1  17000   2516  -  I     4:57PM  0:00.09 /bin/sh /usr/local/pkg/sqpmon.sh
      root    23284  0.0  0.1  43440   2668 u1  Is    3:29PM  0:00.01 login [pam] (login)
      root    23486  0.0  0.1  17000   2636 u1  I     3:29PM  0:00.01 -sh (sh)
      root    23617  0.0  0.1  17000   2524 u1  I+    3:29PM  0:00.00 /bin/sh /etc/rc.initial
      root    17974  0.0  0.1  18676   2308  0  R+    6:17PM  0:00.00 ps aux
      root    32712  0.0  0.1  17000   2620  0  Is    6:14PM  0:00.01 -sh (sh)
      root    32955  0.0  0.1  17000   2528  0  I     6:14PM  0:00.00 /bin/sh /etc/rc.initial
      root    39900  0.0  0.1  17340   3676  0  S     6:14PM  0:00.02 /bin/tcsh
      
      

      It seems that php-fpm now eats up my CPU.
      Where should I start my investigation ?

      Regards,
      Ozy.

      NetPOWER.fr - some opensource stuff for IT people

      1 Reply Last reply Reply Quote 0
      • K Offline
        kejianshi
        last edited by

        Do you have your web access open to the world?

        1 Reply Last reply Reply Quote 0
        • D Offline
          deajan
          last edited by

          I do have web access via WAN on a random port.
          A quick netstat showed me that I'm the only one connected to it.

          NetPOWER.fr - some opensource stuff for IT people

          1 Reply Last reply Reply Quote 0
          • K Offline
            kejianshi
            last edited by

            Close that and use a vpn or SSH.  Opening your web access to the WAN is stupid.

            Don't feel bad - Everyone is stupid before they get smart.

            1 Reply Last reply Reply Quote 0
            • D Offline
              deajan
              last edited by

              Remote web access is already limited to my IP so I guess this isn't the problem here.
              Is there any php profiling available in pfSense ?

              NetPOWER.fr - some opensource stuff for IT people

              1 Reply Last reply Reply Quote 0
              • K Offline
                kejianshi
                last edited by

                Not sure.  I'm often wrong about these things, but me personally.

                I'd limit it to VPN access only.

                1 Reply Last reply Reply Quote 0
                • C Offline
                  cmb
                  last edited by

                  Wouldn't end up with that even if your GUI was open to the world (though that would be a bad idea regardless).

                  Anything in your system log?

                  A truss of what the process is doing might be telling. For instance on the top PID you're showing there:

                  truss -p 83276 -o /root/83276-truss.txt
                  

                  Let that run for 30 seconds or so and hit ctrl-c, then download that file and attach it. Can repeat same for the other php-fpm PIDs if it's like what you're showing there, with 3 separate ones chewing up a good deal of CPU.

                  1 Reply Last reply Reply Quote 0
                  • D Offline
                    deajan
                    last edited by

                    System logs show the following messages like every hour or so:

                    
                    JJun 23 09:31:35	kernel		sonewconn: pcb 0xfffff80061c4d188: Listen queue overflow: 193 already in queue awaiting acceptance (1288 occurrences)
                    Jun 23 09:30:34	kernel		sonewconn: pcb 0xfffff80061c4d188: Listen queue overflow: 193 already in queue awaiting acceptance (523 occurrences)
                    
                    

                    The number 193 and pcb address always stays the same accross all messages.
                    I've tried to isolate which process has the pcb address with lsof | grep 0xfff…. but I didn't get any result.
                    I've tried to increase kern.ipc.somaxconn from 128 to 512 but this doen't change cpu usage.

                    The php-fpm processes spawn and exit after like 30 seconds, so truss output might be shorter than 30 seconds.
                    Attached are two truss outputs.

                    PS: as for the open web configurator access, I worked on a tap VPN bridge past days, but in the meantime I had to be able to configure it :)

                    truss.73254.log.tar.gz
                    truss.13089.log.tar.gz

                    NetPOWER.fr - some opensource stuff for IT people

                    1 Reply Last reply Reply Quote 0
                    • C Offline
                      cmb
                      last edited by

                      It's captive portal that's getting beat up there. Guessing you don't have a "Maximum concurrent connections" limit set in your CP, and some device is on that network, not logged into the portal, with an update checker or something running in the background that's causing numerous non-stop requests to hit CP.

                      1 Reply Last reply Reply Quote 0
                      • D Offline
                        deajan
                        last edited by

                        Thank you for the insight.
                        I renamed the thread to something more explicit. Can a forum admin move it to captiveportal perhaps ?

                        After knowing what I was searching for, I isolated the problem to one device that tries to access 50-70 times a second an URL.

                        tail /var/log/nginx.log

                        
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /index.php?zone=lan&redirurl=http%3A%2F%2Fweatherblink.wdgserv.com%2Fweatherblink%2Flookup%2FLyon%2C+France HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /index.php?zone=lan&redirurl=http%3A%2F%2Fweatherblink.wdgserv.com%2Fweatherblink%2Flookup%2FLyon%2C+France HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /index.php?zone=lan&redirurl=http%3A%2F%2Fweatherblink.wdgserv.com%2Fweatherblink%2Flookup%2FLyon%2C+France HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        Jun 24 10:53:09 pfsensebox pfsensebox.company.local nginx: 172.16.2.238 - - [24/Jun/2016:10:53:09 +0200] "GET /weatherblink/lookup/Lyon,%20France HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
                        .......
                        
                        

                        The setup I have is an open internet access for 200 student flats, where the user account is created on the fly by the CP page.
                        Everyone can bring as many devices as they want, that's the demand of the customer.

                        Is there any way I can limit CP demands per IP or MAC base in order to mitigate some random pri*** using the worst software ever written to flood my pfsense CP ?

                        I was thinking that between 2.3.1-1 and 2.3.1-5, the keepalive_timeout for the CP nginx instance was disabled (#6421). Maybe this triggered the new behavior ?
                        Also, maybe lowering the value of limit_conn addr could help.

                        Any thoughts ?

                        Regards,
                        Ozy.

                        NetPOWER.fr - some opensource stuff for IT people

                        1 Reply Last reply Reply Quote 0
                        • D Offline
                          deajan
                          last edited by

                          OMG ! Reading through the code in system.inc, I realized that limit_conn addr = "Maximum concurrent connections" in the CP web interface.
                          I lowered the default value (10) to 3. Will see what happens.

                          Any idea what's a "good" value for the max concurrent connections value ?

                          NetPOWER.fr - some opensource stuff for IT people

                          1 Reply Last reply Reply Quote 0
                          • C Offline
                            cmb
                            last edited by

                            Good point, didn't think about it but I bet the keepalive timeout being disabled now probably makes the simultaneous connections significantly less effective than it used to be. Those stupid apps that are doing repeated calls like that are likely using some kind of library to do it that will utilize the keepalive. When the keepalive is disabled, they'll issue a new request every time and close out the former immediately, so if requests are issued sequentially you can rack up a huge load with something really persistent.

                            How many is appropriate depends on what's in your portal page. If it's just the HTML page with no images, no CSS, js, etc. - nothing other than one page to fetch, then 2-3 is probably a fine limit.

                            1 Reply Last reply Reply Quote 0
                            • DerelictD Offline
                              Derelict LAYER 8 Netgate
                              last edited by

                              Maybe a keepalive of like 2-5 seconds would be better? Enough to reap the benefits in these rapid-fire cases but should work for all but the fastest CP logins?

                              If it is establishing a new session every time it won't matter anyway. Have to see a tcpdump.

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • C Offline
                                cmb
                                last edited by

                                Guessing since it started being noticeable post-upgrade it was probably obeying the keepalive. Disabling it is definitely nicer for purposes of redirecting users instantaneously in all cases. Maybe a smaller keepalive would be more appropriate, something like 2 seconds possibly. There are also rate limiting options in nginx which might be more appropriate instead, or maybe in addition to. Something like 1 request/sec, though for users who have CSS, images, etc. on the portal page that's too low. Maybe make that a configurable option as well.

                                1 Reply Last reply Reply Quote 0
                                • D Offline
                                  deajan
                                  last edited by

                                  My CP page is kinda complex with mysql, css, bootstrap and a lot of php code.

                                  Setting maximum concurrent connections from 10 to 3 decreased cpu usage from 100 to 30%.
                                  Setting keepalive 2 gave an additionnal cpu drop, usage is now 7%-15%.

                                  Maybe setting keepalive to 2 is indeed a good solution for next release ?

                                  Regards,
                                  Ozy.

                                  NetPOWER.fr - some opensource stuff for IT people

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.