Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    Could not connect to /var/run/php-fpm.socket

    Scheduled Pinned Locked Moved General pfSense Questions
    11 Posts 4 Posters 1.5k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T Offline
      thebear
      last edited by

      Hi all

      installed pfSense to my NUC7i3BNK, somehow my pfSense webGUI stops. SSH is still available, and to regain access I logon and choose number 16 -> Restart PHP-FPM

      How could we find the root cause, anybody willing to help?

      Jul 18 20:55:18 pfSense nginx: 2021/07/18 20:55:18 [error] 45051#100136: *794 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 172.16.X.50, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "172.16.Y.1:8443"

      Who is the upstream, and why its relevant for local mgmt?

      1 Reply Last reply Reply Quote 0
      • A Offline
        andrewinhawaii
        last edited by

        I'm getting the same thing with version 2.8.1-RELEASE running on an HP T640 thin client.

        I believe that I saw out of the corner of my eye CPU usage on the dashboard, which is normally 0% - 1%, pop up to 65% and then when I tried to refresh, I got the "50x Error" page. I had to SSH in and select "Restart PHP-FPM". The system general log file was full of "nginx [Date Time] XXXXX#YYYYYY: *ZZZZZZ connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: [my internal IP address], server: , request: "POST /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:"" messages.

        This all started out of the blue when I moved my pfSense installation from a failing HP T630 thin client to a newly prepared T640 including a fresh install of 2.8.1-RELEASE. I had saved the old configuration from the T630 running 2.8.1-RELEASE and installed it on the the new hardware. Previously, pfSense had been extremely reliable, and I have been running versions of it for more than a decade.

        GertjanG 1 Reply Last reply Reply Quote 0
        • GertjanG Offline
          Gertjan @andrewinhawaii
          last edited by

          @andrewinhawaii

          The pfSense GUI uses nginx, a web server, the post the 'site' that generates the GUI.
          What you see on the screen, in your web browser, doesn't exists as files in the pfSense disk.
          The web server reads the files ( you can find them here : /usr/local/www/ ) and it was told that if it finds '.php' files, it has to send these files, PHP scripts actually, to : see the nginx web server config file : /var/etc/nginx-webConfigurator.conf - the " location ~ .php$ {" part, to :

          fastcgi_pass   unix:/var/run/php-fpm.socket;

          That 'socket' is the place where the PHP interpreter listens for PHP scripts, executes them, and send the result back to the web server. The result will be pure html, and that's what your web browser can understand.
          This is how 75 % (?) of all 2 trillion web servers on Internet work. Pretty standard.

          php (the process php-fpm) can die = stop executing. In that case the web server starts to produce the error log line you've shown : it can't 'talk' the the php process anymore. The 'socket' is dead.
          'php' can die = stop executing - because : a memory issue ? Be ware that php doesn't use all avaible memory, but a reserved block, see : System > Advanced > Miscellaneous under PHP Settings.
          If the system goes low on memory, the OS can elect php for it to kicked out to free up memory.
          Or you've asked PHP to do more then it can handle : showing huge logs files, or, if you use pfBlockerng, loads a big number of DNSBL feeds.
          These are just examples.
          The system logs most probably contains an earlier log message from the PHP process where you can see that it announces that it's stopping, and with some luck the reason. Up to you to remove this reason.

          No "help me" PM's please. Use the forum, the community will thank you.

          1 Reply Last reply Reply Quote 0
          • A Offline
            andrewinhawaii
            last edited by

            OK, so it's not that php is sometimes dying sometimes. It's dying all the time, some times minutes after a reboot. It's not a memory problem: they system has 16GB RAM, and I used the advanced settings to crank up the PHP allocated memory from 512MB to 2GB. If it needs more than 2GB, Lord help me.

            Here is a process listing when things are good:

            [2.8.1-RELEASE][admin@fw.bananas.com]/root: ps -auxwww | fgrep php
root   29524   0.0  0.0  13980  2476  -  Is   15:15      0:00.00 /usr/local/bin/minicron 300 /var/run/ipsec_keepalive.pid /usr/local/bin/ipsec_keepalive.php
root   30036   0.0  0.0  13980  2496  -  I    15:15      0:00.02 minicron: helper /usr/local/bin/ipsec_keepalive.php  (minicron)
root   43176   0.0  0.3 111208 37728  -  Ss   11:36      0:00.00 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root   43332   0.0  0.3 111208 37748  -  S    11:36      0:00.00 php-fpm: pool nginx (php-fpm)
root   43608   0.0  0.3 111208 37752  -  S    11:36      0:00.00 php-fpm: pool nginx (php-fpm)
root   52054   0.0  0.0  14076  2676  0  S+   11:37      0:00.00 fgrep php

            And here is a process listing when things have gone South:

            [2.8.1-RELEASE][admin@fw.bananas.com]/root: ps -auxwww | fgrep php
root    4303   0.0  0.4 117352 52944  -  I    12:07      0:00.01 php-fpm: pool nginx (php-fpm)
root    4800   0.0  0.4 117352 52948  -  I    12:07      0:00.01 php-fpm: pool nginx (php-fpm)
root    5059   0.0  0.4 117352 52948  -  I    12:07      0:00.01 php-fpm: pool nginx (php-fpm)
root   29524   0.0  0.0  13980  2476  -  Is   15:15      0:00.00 /usr/local/bin/minicron 300 /var/run/ipsec_keepalive.pid /usr/local/bin/ipsec_keepalive.php
root   30036   0.0  0.0  13980  2496  -  I    15:15      0:00.02 minicron: helper /usr/local/bin/ipsec_keepalive.php  (minicron)
root   43176   0.0  0.3 111208 37732  -  Ss   11:36      0:00.13 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root   43332   0.0  0.4 164984 59444  -  I    11:36      0:04.12 php-fpm: pool nginx (php-fpm)
root   43608   0.0  0.4 164920 58708  -  I    11:36      0:00.92 php-fpm: pool nginx (php-fpm)
root   59372   0.0  0.4 164920 59152  -  I    11:39      0:00.94 php-fpm: pool nginx (php-fpm)
root   71693   0.0  0.4 164920 58768  -  I    11:49      0:01.98 php-fpm: pool nginx (php-fpm)
root   80159   0.0  0.4 164920 59124  -  I    11:37      0:03.67 php-fpm: pool nginx (php-fpm)
root   62566   0.0  0.0  14076  2680  0  S+   12:59      0:00.00 fgrep php

            It seems that a bunch of extra threads were created about 30 minutes after boot. I can't tell you exactly when the web interface became non-responsive but if I were a betting man, I'd say it was at about 30 minutes after boot.

            This is what typically happens (immediately after the reboot above) from /var/log/nginx.log:

            May 24 11:35:54 fw nginx: 10.0.128.236 - - [24/May/2026:11:35:54 -1000] "GET / H
TTP/1.1" 502 1997 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:13
9.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "GET / H
TTP/1.1" 200 20463 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:1
39.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "GET /ve
ndor/font-awesome/webfonts/fa-solid-900.woff2 HTTP/1.1" 200 156496 "http://fw.16
paws.com/vendor/font-awesome/css/all.min.css?v=1756397406" "Mozilla/5.0 (X11; Li
nux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "GET /ve
ndor/font-awesome/webfonts/fa-regular-400.woff2 HTTP/1.1" 200 25452 "http://fw.1
6paws.com/vendor/font-awesome/css/all.min.css?v=1756397406" "Mozilla/5.0 (X11; L
inux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "GET /ap
ple-touch/apple-touch-icon-ipad-retina-152x152-precomposed.png HTTP/1.1" 200 726
0 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100
101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "GET /fa
vicon-32x32.png HTTP/1.1" 200 1842 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Li
nux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "POST /w
idgets/widgets/nut_status.widget.php HTTP/1.1" 200 184 "http://fw.bananas.com/" "
Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 11:37:08 fw nginx: 10.0.128.236 - - [24/May/2026:11:37:08 -1000] "POST /w
idgets/widgets/disks.widget.php HTTP/1.1" 200 505 "http://fw.bananas.com/" "Mozil
la/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"

            About 30 minutes later:

            May 24 12:07:04 fw nginx: 10.0.128.236 - - [24/May/2026:12:07:04 -1000] "POST /g
etstats.php HTTP/1.1" 200 141 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x
86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 12:10:05 fw nginx: 2026/05/24 12:10:05 [error] 79421#100255: *19669 upstr
eam timed out (60: Operation timed out) while reading response header from upstr
eam, client: 10.0.128.236, server: , request: "POST /ifstats.php HTTP/1.1", upst
ream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.bananas.com", referrer:
 "http://fw.bananas.com/"
May 24 12:10:05 fw nginx: 10.0.128.236 - - [24/May/2026:12:10:05 -1000] "POST /i
fstats.php HTTP/1.1" 504 1997 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x
86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 12:10:09 fw nginx: 2026/05/24 12:10:09 [error] 79421#100255: *19672 upstr
eam timed out (60: Operation timed out) while reading response header from upstr
eam, client: 10.0.128.236, server: , request: "POST /getstats.php HTTP/1.1", ups
tream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.bananas.com", referrer
: "http://fw.bananas.com/"
May 24 12:10:09 fw nginx: 10.0.128.236 - - [24/May/2026:12:10:09 -1000] "POST /getstats.php HTTP/1.1" 504 1997 "http://fw.bananas.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0"
May 24 12:10:11 fw nginx: 2026/05/24 12:10:11 [error] 79421#100255: *19674 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.0.128.236, server: , request: "POST /widgets/widgets/nut_status.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.bananas.com", referrer: "http://fw.bananas.com/"

            Is there any other diagnostic information that I could provide to help figure out what is going on? Another, possibly related, possibly unrelated, issue is that dnsmasq just sometimes stops working too. This causes web pages to stop loading, including the dashboard if I am using the canonical name FW, but not if I am using its internal IP address of 10.0.0.1. All of this instability began after I moved from an HP T630 to a T640, and added a second WAN provider. WAN and WAN2 are in a gateway group as Tier 1 and Tier 2 for failover.

            Thanks,

            Andrew

            GertjanG 1 Reply Last reply Reply Quote 0
            • GertjanG Offline
              Gertjan @andrewinhawaii
              last edited by

              @andrewinhawaii

              PHP died during the 30 minutes :

              db730ea0-98b5-4680-860d-724a46a10c7f-image.png

              Did 'php' or php-fpm log something during that time ?

              No "help me" PM's please. Use the forum, the community will thank you.

              1 Reply Last reply Reply Quote 0
              • A Offline
                andrewinhawaii
                last edited by

                Checking /var/log/system.log I came across this:

                May 24 11:35:54 fw nginx: 2026/05/24 11:35:54 [error] 79078#100237: *19662 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 10.0.128.236, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "fw.bananas.com", referrer: "http://fw.bananas.com/"
May 24 11:35:54 fw kernel: sonewconn: pcb 0xfffff80159188280 (local:/var/run/php-fpm.socket): Listen queue overflow: 193 already in queue awaiting acceptance (6 occurrences), euid 0, rgid 0, jail 0
May 24 11:36:50 fw rc.php-fpm_restart[42671]: >>> Restarting php-fpm
May 24 11:36:50 fw check_reload_status[44059]: check_reload_status is starting.
May 24 11:46:15 fw check_reload_status[44059]: Linkup starting ue1
May 24 11:46:15 fw check_reload_status[44059]: Linkup starting ue1
May 24 11:46:15 fw kernel: ue1: link state changed to DOWN
May 24 11:46:15 fw kernel: ue1: 3 link states coalesced
May 24 11:46:15 fw kernel: ue1: link state changed to UP
May 24 11:46:21 fw php-fpm[43332]: /widgets/widgets/traffic_graphs.widget.php: Configuration Change: admin@10.0.128.236 (Local Database): Updated traffic graphs widget settings via dashboard.
May 24 11:46:21 fw check_reload_status[44059]: Syncing firewall
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to DOWN
May 24 12:07:04 fw kernel: ue1: 3 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw kernel: ue1: 2 link states coalesced
May 24 12:07:04 fw kernel: ue1: link state changed to UP
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:04 fw check_reload_status[44059]: Linkup starting ue1
May 24 12:07:05 fw check_reload_status[44059]: rc.newwanip starting ue1
May 24 12:10:05 fw nginx: 2026/05/24 12:10:05 [error] 79421#100255: *19669 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.0.128.236, server: , request: "POST /ifstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.bananas.com", referrer: "http://fw.bananas.com/"
May 24 12:10:09 fw nginx: 2026/05/24 12:10:09 [error] 79421#100255: *19672 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.0.128.236, server: , request: "POST /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.bananas.com", referrer: "http://fw.bananas.com/"

                At the top (11:35:54) appears to be a symptom of the problem before I restarted php-fpm. Then at 11:36:50, you can see where I restarted php-fpm. Then at 12:07:04 seems to be where all hell breaks loose. And at 12:10:05 we can see the timeouts begin.

                Is there another log file specifically for php-fpm that I can check?

                Thanks,

                Andrew

                GertjanG 1 Reply Last reply Reply Quote 0
                • GertjanG Offline
                  Gertjan @andrewinhawaii
                  last edited by

                  @andrewinhawaii

                  Every time a hardware link, like eu1, goes down, some OS code and PHP scripts is executed, to keep the state of the pfSense 'machine' up to date.
                  When the link comes up, a lot of - like way more - code is executed, and among that, for a lot, is PHP called.

                  Switches, hubs, access points etc are fine with devices that come and go, but for a router/firewall, flapping interfaces is bad (or worse).
                  Can you make this eu1 interface stop doing what it is doing ? What is it connected to , and why is is chain gunning the interface like that ?

                  What pfSense packages are you using ?

                  No "help me" PM's please. Use the forum, the community will thank you.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    Mmm, something is using all the available PHP processes.

                    And, yes, ue1 looks like it's flapping which will be triggering a lot of PHP scripts.

                    The existence of ue1 implies you also have ue0. USB NICs are generally not recommended and can be unstable like that. Can you move those other NICs?

                    1 Reply Last reply Reply Quote 0
                    • GertjanG Gertjan referenced this topic on
                    • A Offline
                      andrewinhawaii
                      last edited by

                      Info: I am running 2.8.1-RELEASE, WAN is a Nokia ONT from my local telcom monopoly, and WAN2 is a DOCSIS 3.1 cable modem from my local cable monopoly. Currently the cable modem is not talking to the world and I will deal with that when I have the time.

                      I got up this morning to find everything on my LAN to be nonresponsive. I ssh'ed into the firewall and found this:

                      [2.8.1-RELEASE][admin@fw.1bananas.com]/root: top
last pid: 53705;  load averages:    2.00,    2.92,    2.94                   up 1+02:20:03  11:33:10
68 processes:  3 running, 65 sleeping
CPU: 16.7% user, 16.7% nice, 22.1% system,  0.1% interrupt, 44.5% idle
Mem: 332M Active, 171M Inact, 938M Wired, 12G Free
ARC: 207M Total, 30M MFU, 167M MRU, 636K Anon, 4424K Header, 5292K Other
     170M Compressed, 2194M Uncompressed, 12.89:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  496 root          1 143   20   305M   250M CPU0     0 532:55  91.76% check_reload_status
54723 root          1 121    0    14M  3424K CPU3     3 510:23  85.39% syslogd
40603 root          1  56    0   115M    48M lockf    1   0:50   1.20% php-fpm
14885 root          1  68    0   115M    48M nanslp   0   0:58   0.62% php-fpm
10309 root          1  20    0    24M    11M select   1   0:01   0.54% sshd
93847 root          1  37    0   115M    48M lockf    3   0:51   0.50% php-fpm
56650 root          1  68    0   115M    48M nanslp   0   0:49   0.50% php-fpm
99534 root          1  55    0   115M    48M lockf    3   0:57   0.42% php-fpm
41090 root          1  68    0   115M    48M nanslp   0   0:53   0.16% php-fpm
 1236 root          1  20    0    15M  5056K select   2   1:23   0.10% devd
82926 root          1  39    0    14M  2608K kqread   2   0:00   0.09% dhcpleases
....
[2.8.1-RELEASE][admin@fw.1bananas.com]/root: wc /var/log/system.log
  715013 7150609 64353898 /var/log/system.log
[2.8.1-RELEASE][admin@fw.bananas.com]/root: tail /var/log/system.log
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw check_reload_status[496]: Could not connect to /var/run/php-fpm.socket
May 27 11:34:54 fw dhcpleases[31681]: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) cannot be read, No such file or directory.

                      I'll bet the CPU was running hot for hours.

                      The existence of ue1 implies you also have ue0. USB NICs are generally not recommended and can be unstable like that. Can you move those other NICs?

                      The HP T640 thin client comes with a single Realtek RTL8111HSH GbE adapter. I am using it for the LAN interface. I have two USB 3.0 GbE adapters for WAN and WAN2 on USB 3.1 Gen 1 ports. Yes, they are suboptimal but that's the price for a low power, fanless AMD Ryzen R1505G Zen1, 16GB DDR4, and 128GB NVMe system for about $30.

                      I put the two WAN ports on USB adapters because I don't trust current FBSD to always enumerate them the same way every reboot. Worst case enumeration would swap them and end up with my LAN interface on a public fixed IP network.

                      I don't know what is triggering this behavior, but it is forcing me to constantly tend the machine, and reboot several times per day. Which is quite a let down from a system that I've had running for years without a care.

                      I just swapped out the two (different) USB GbE adapters to 2 identical adapters and the whole thing fell over. WAN was up, WAN was down. Ping shows no route to the firewall on the LAN while I am ssh'ed in from the same host. Then I get pings that die due to TTL that look like this:

                      From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1: icmp_seq=186 Redirect Host(New nexthop: 10.0.0.1)
From 10.0.0.1 icmp_seq=186 Time to live exceeded

                      I must have rebooted about a dozen times before I got something stable enough for the time being.

                      So the question is: what can be done to prevent this behavior? Or is this expected? Yes, USB network if's are suboptimal. I can expect and accept that maybe you won't get the full 1Gb throughput but I'm not expecting that from my two ISPs anyway. USB GbE if's are not new technology and should be fairly reliable. Afaik they are not supposed to cause the whole system to become wildly unstable, fall over in multiple ways, and be generally unusable. There seems to be some issues that are handled poorly and daemons allowed to run wild taking 100% of CPU time. Neither of these seems like a desirable state.

                      Andrew

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S Offline
                        stephenw10 Netgate Administrator
                        last edited by

                        Unfortunately there's not much you can do with flapping USB NICs. I would probably use re0 as WAN and one usb NIC as LAN and see if that prevents the issue.

                        Otherwise I'd use VLANs and a suitable switch to separate the interfaces in a single NIC.

                        1 Reply Last reply Reply Quote 0
                        • A Offline
                          andrewinhawaii
                          last edited by

                          OK, I now understand the nature of the problem. See PR 252165.

                          Ali Abdallah 2024-11-13 21:26:53 UTC
                          The usb process/request code is completely broken when it comes to different threads sharing ue_lock (ioctl versus uether tick), the usb request code releases the acquired lock, making ioctl thread waiting for the same lock to be scheduled for execution, lovely!

                          To be more precise.

                          ioctly waiting for ue_lock, which is acquired by ue_tick, but then the tick code will call at some point usbd_do_request_flags, which will release the lock at the beginning (and then acquire it later), but in the meantime we have our "cute" ioctl thread waken-up only to report wrong media_status value!

                          vxasxfepbikrfqdruz 2022-10-22 04:06:44 UTC
                          Yeah, I was pretty surprised as well that this is entirely broken despite several USB to ethernet adapters being listed as supported by FreeBSD. This also affects pfSense and opnSense which you think would generate some attention about this, but they just seem to recommend not using USB adapters at all.

                          Ali Abdallah 2026-03-16 16:00:12 UTC
                          I got excited for a moment when I saw a commit referencing this bug, I thought that we have finally a solution, but unfortunately this is just a workaround, it is not a resolution for the issue, and it is only implemented for if_ure, other usb network drivers suffers also from this bug (if_muge, if_axge, if_rue, etc...)

                          TBH after all these years, I was expecting a real solution, at the usb request code level (see comment 38). If a workaround is to be implemented, there are easier solutions (see comment 3, patch https://bugs.freebsd.org/bugzilla/attachment.cgi?id=221274&action=diff) which needs to be extended also for other phys at the mii bus level.

                          So the MII layer driver was, is, and will be completely broken in FreeBSD.

                          And the number of failure modes from a driver bug seems unbounded. For example, after a dozen or so reboots yesterday, I had a stable system for about a day. Until I didn't. PFP-FPM failure was a sign, so I restarted it. Still no joy. No DNS response. I tried killing and restarting dnsmasq but that did not help. Rebooting showed dnsmasq running, but still no DNS service immediately. Again I rebooted about a half dozen times until I finally got DNS service. I guess the best I can hope for is another day of operation until I have to repeat.

                          Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
0.0.0.0            xxx.yyy.zzz.1      UGS             ue1
8.8.8.8            xxx.yyy.zzz.1      UGHS            ue1
10.0.0.0/16        link#1             U               re0
10.0.0.1           link#3             UHS             lo0
10.1.0.0/16        10.0.0.1           UGS             re0
xxx.yyy.zzz.0/23   link#7             U               ue1
xxx.yyy.zzz.247    link#3             UHS             lo0
127.0.0.1          link#3             UH              lo0
[2.8.1-RELEASE][admin@fw.bananas.com]/root: nslookup orange
;; communications error to 127.0.0.1#53: timed out
;; communications error to 127.0.0.1#53: timed out
;; communications error to 127.0.0.1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 10.1.0.1#53: timed out
;; no servers could be reached

[2.8.1-RELEASE][admin@fw.bananas.com]/root: nslookup www.google.com
;; communications error to 127.0.0.1#53: timed out
;; communications error to 127.0.0.1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 10.1.0.1#53: timed out
;; no servers could be reached

[2.8.1-RELEASE][admin@fw.bananas.com]/root: ps -auxww | fgrep dns
nobody 34809   0.5  0.0  16644  5080  -  S    10:51   0:00.01 /usr/local/sbin/dnsmasq -C /dev/null --dhcp-hostsfile=/etc/hosts --no-resolv --server=10.1.0.1 --server=8.8.8.8 --server=1.1.1.1 --server=4.4.4.4 --strict-order --dns-forward-max=5000 --cache-size=10000 --local-ttl=1
root   46981   0.0  0.0  13988  2612  -  Ss   10:51   0:00.00 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d bananas.com -p /var/run/dnsmasq.pid -h /etc/hosts
root   92820   0.0  0.0  22908  3596  -  Is   10:50   0:00.00 /usr/local/sbin/filterdns -p /var/run/filterdns-ipsec.pid -i 60 -c /var/etc/ipsec/filterdns-ipsec.hosts -d 1
root   47137   0.0  0.0  14076  2676  0  S+   10:51   0:00.00 fgrep dns
[2.8.1-RELEASE][admin@fw.bananas.com]/root:

                          You can't tell me that this is all because the USB network drivers are "flappy". This is house of cards that falls down due to unhandled errors. And is blamed on the USB if instead of correcting.

                          Andrew

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.
                          Privacy Policy · Cookie Policy