Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.3 Lockup with no packages

    Scheduled Pinned Locked Moved
    Problems Installing or Upgrading pfSense Software
    10
    61
    10.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      SoloIT
      last edited by

      Firewall box with em NIC and modified kernel is up and running. It's seems to be routing traffic as normal. I should know by morning how it's holding up. In the past it has crashed during my remote back which moves a lot of data.

      I will keep everyone posted. Again thank you all for your help.

      1 Reply Last reply Reply Quote 0
      • S
        SoloIT
        last edited by

        13.5 hours and no crashes. I did have odd stuff in my system.log. Any ideas?

        
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *371 open() "/usr/local/www/redir/cgi-bin/ajaxmail" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /redir/cgi-bin/ajaxmail HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *370 open() "/usr/local/www/fcgi-bin/performance.fcgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /fcgi-bin/performance.fcgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *369 open() "/usr/local/www/fcgi-bin/dispatch.fcgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /fcgi-bin/dispatch.fcgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *368 open() "/usr/local/www/das/cgi-bin/session.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /das/cgi-bin/session.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *367 open() "/usr/local/www/wingame.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /wingame.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *366 open() "/usr/local/www/webscr" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /webscr HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *365 open() "/usr/local/www/webproc" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /webproc HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *364 open() "/usr/local/www/verify.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /verify.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *363 open() "/usr/local/www/traffic/process.fcgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /traffic/process.fcgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *362 open() "/usr/local/www/top/out" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /top/out HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *361 open() "/usr/local/www/tjcgi1" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /tjcgi1 HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:52	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:52 [error] 27529#0: *360 open() "/usr/local/www/te/o.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /te/o.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *359 open() "/usr/local/www/start" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /start HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *358 open() "/usr/local/www/sse.dll" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /sse.dll HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *357 open() "/usr/local/www/spcnweb" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /spcnweb HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *356 open() "/usr/local/www/search.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /search.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *355 open() "/usr/local/www/rshop.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /rshop.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *354 open() "/usr/local/www/readmsg" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /readmsg HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *353 open() "/usr/local/www/rbaccess/rbunxcgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /rbaccess/rbunxcgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *352 open() "/usr/local/www/rbaccess/rbcgi3m01" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /rbaccess/rbcgi3m01 HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *351 open() "/usr/local/www/passremind" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /passremind HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *350 open() "/usr/local/www/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *349 open() "/usr/local/www/openwebmail/openwebmail-main.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /openwebmail/openwebmail-main.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *348 open() "/usr/local/www/navega" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /navega HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *347 open() "/usr/local/www/msglist" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /msglist HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *346 open() "/usr/local/www/mainsrch" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /mainsrch HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *345 open() "/usr/local/www/mainmenu.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /mainmenu.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *344 open() "/usr/local/www/logout" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /logout HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *343 open() "/usr/local/www/logout" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /logout HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *342 open() "/usr/local/www/login" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /login HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *341 open() "/usr/local/www/login.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /login.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *340 open() "/usr/local/www/link" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /link HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *339 open() "/usr/local/www/krcgistart" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /krcgistart HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *338 open() "/usr/local/www/krcgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /krcgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *337 open() "/usr/local/www/index" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /index HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *336 open() "/usr/local/www/index.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /index.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *335 open() "/usr/local/www/ib/301_start.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /ib/301_start.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *334 open() "/usr/local/www/hslogin.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /hslogin.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *333 open() "/usr/local/www/hotspotlogin.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /hotspotlogin.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *332 open() "/usr/local/www/getattach" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /getattach HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *331 open() "/usr/local/www/frame_html" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /frame_html HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *330 open() "/usr/local/www/findweather/hdfForecast" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /findweather/hdfForecast HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *329 open() "/usr/local/www/findweather/getForecast" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /findweather/getForecast HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *328 open() "/usr/local/www/fg.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /fg.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *327 open() "/usr/local/www/crtr/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /crtr/out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *326 open() "/usr/local/www/clicks.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /clicks.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *325 open() "/usr/local/www/click.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /click.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *324 open() "/usr/local/www/br5.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /br5.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *323 open() "/usr/local/www/bp_revision.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /bp_revision.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *322 open() "/usr/local/www/bbs/postshow.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /bbs/postshow.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *321 open() "/usr/local/www/bbs/postlist.pl" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /bbs/postlist.pl HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *320 open() "/usr/local/www/auth" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /auth HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *319 open() "/usr/local/www/atx/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /atx/out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *318 open() "/usr/local/www/atc/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /atc/out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *317 open() "/usr/local/www/at3/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /at3/out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *316 open() "/usr/local/www/arr/index.shtml" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /arr/index.shtml HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27254#0: *315 open() "/usr/local/www/ajaxmail" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /ajaxmail HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *314 open() "/usr/local/www/a2/out.cgi" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /a2/out.cgi HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *313 open() "/usr/local/www/cgi-bin/webproc" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /cgi-bin/webproc?getpage=/../../etc/passwd&var:language=en_us&var:page=* HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:51	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:51 [error] 27529#0: *312 open() "/usr/local/www/rom-0" failed (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /rom-0 HTTP/1.1", host: "10.10.1.1"
Apr 26 06:54:25	bhamfirewall.warrenmfg.local		nginx: 2016/04/26 06:54:25 [error] 27529#0: *311 "/usr/local/www/HNAP1/index.html" is not found (2: No such file or directory), client: 10.10.1.177, server: , request: "GET /HNAP1/ HTTP/1.1", host: "10.10.1.1"
        1 Reply Last reply Reply Quote 0
        • M
          mer
          last edited by

          Well, that's good news on the modified kernel.  Are 10.10.1.1 and 10.10.1.177 IPs in your network?  If you ssh/console to the box, ls /usr/local/www are any of those files/directories present?  Should they be?

          1 Reply Last reply Reply Quote 0
          • PerforadoP
            Perforado Rebel Alliance
            last edited by

            10.10.1.177 seems to be infected with some Malware i guess  :o

            1 Reply Last reply Reply Quote 0
            • S
              SoloIT
              last edited by

              10.10.1.1 is the firewall and 10.10.1.177 is client. I'm so tired of looking at the log files, I total glossed over the accessing IP. I'll look into it and see if there is any malware on the machine.

              I did not look for all the files/folders, but the dozen or so I checked do not exist.

              Thanks for the second set of eyes.

              1 Reply Last reply Reply Quote 0
              • C
                cmb
                last edited by

                That 10.10.1.177 indeed looks to be compromised, or possibly running a vulnerability assessment tool but that list seems to match known malware. Something on that machine is scanning looking for common vulnerable pages.

                1 Reply Last reply Reply Quote 0
                • S
                  SoloIT
                  last edited by

                  Thanks I am looking into that client. It's used by a part time person, so it explains why I've not seen it before.

                  Good news, the patch is still holding up. No crashes. I'm planning to roll the patch to the other firewalls tonight.

                  This weekend I'll also change back to my other hardware platform for the one odd location using the igb NIC rather than then em that most of my firewalls use.

                  1 Reply Last reply Reply Quote 0
                  • S
                    SoloIT
                    last edited by

                    I did update all my other firewall's with the patch yesterday evening.
                    This morning, the first test box I had installed the patch on encountered the same watchdog error:

                    
Apr 27 01:24:22 bhamfirewall kernel: em2: Watchdog timeout -- resetting
Apr 27 01:24:22 bhamfirewall kernel: em2: link state changed to DOWN
Apr 27 01:24:22 bhamfirewall check_reload_status: Linkup starting em2
Apr 27 01:24:23 bhamfirewall php-fpm[13740]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (10.10.1.1 )
Apr 27 01:24:25 bhamfirewall kernel: em2: link state changed to UP
Apr 27 01:24:25 bhamfirewall check_reload_status: Linkup starting em2
Apr 27 01:24:26 bhamfirewall php-fpm[13740]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (10.10.1.1 )
Apr 27 01:24:26 bhamfirewall check_reload_status: rc.newwanip starting em2
Apr 27 01:24:27 bhamfirewall php-fpm[13740]: /rc.newwanip: rc.newwanip: Info: starting on em2.
Apr 27 01:24:27 bhamfirewall php-fpm[13740]: /rc.newwanip: rc.newwanip: on (IP address: 10.10.1.1) (interface: LAN[lan]) (real interface: em2).
Apr 27 01:24:27 bhamfirewall check_reload_status: Reloading filter
Apr 27 01:24:28 bhamfirewall xinetd[11505]: Starting reconfiguration
Apr 27 01:24:28 bhamfirewall xinetd[11505]: Swapping defaults
Apr 27 01:24:28 bhamfirewall xinetd[11505]: readjusting service 6969-udp
Apr 27 01:24:28 bhamfirewall xinetd[11505]: Reconfigured: new=0 old=1 dropped=0 (services)
Apr 27 01:25:46 bhamfirewall kernel: em2: Watchdog timeout -- resetting
Apr 27 01:25:46 bhamfirewall kernel: em2: link state changed to DOWN
Apr 27 01:25:46 bhamfirewall check_reload_status: Linkup starting em2
Apr 27 01:25:47 bhamfirewall php-fpm[13740]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (10.10.1.1 )
Apr 27 01:25:49 bhamfirewall check_reload_status: Linkup starting em2
Apr 27 01:25:49 bhamfirewall kernel: em2: link state changed to UP
Apr 27 01:25:50 bhamfirewall php-fpm[13740]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (10.10.1.1 )
Apr 27 01:25:50 bhamfirewall check_reload_status: rc.newwanip starting em2
Apr 27 01:25:51 bhamfirewall php-fpm[13740]: /rc.newwanip: rc.newwanip: Info: starting on em2.
Apr 27 01:25:51 bhamfirewall php-fpm[13740]: /rc.newwanip: rc.newwanip: on (IP address: 10.10.1.1) (interface: LAN[lan]) (real interface: em2).
Apr 27 01:25:51 bhamfirewall check_reload_status: Reloading filter
Apr 27 01:25:52 bhamfirewall xinetd[11505]: Starting reconfiguration
Apr 27 01:25:52 bhamfirewall xinetd[11505]: Swapping defaults
Apr 27 01:25:52 bhamfirewall xinetd[11505]: readjusting service 6969-udp
Apr 27 01:25:52 bhamfirewall xinetd[11505]: Reconfigured: new=0 old=1 dropped=0 (services)

                    Above is the system.log from when it started. There is nothing before. Once it starts, this set of error messages just keeps repeating until I reset the firewall. Since it was the LAN port acting up, I could still ping the firewall from the WAN and establish an OpenVPN connection. Also, my IPSec tunnels remained running.

                    I was really hoping the patch fixed it the problem as it was running for over a day and typically locked up before then. Suggestions are appreciated.

                    1 Reply Last reply Reply Quote 0
                    • M
                      mer
                      last edited by

                      Anything happening at about that timeframe, backups, remote transfer of lots of data?  One of the other users of the modified kernel reported a problem after awhile, during a transfer of large amount of data.

                      In all the other cases of you having an issue, was the WAN side still alive but the LAN side dead//stuck

                      1 Reply Last reply Reply Quote 0
                      • S
                        SoloIT
                        last edited by

                        Yes, a lot of data was being moved as a back to a remote site was in progress.

                        In the other cases, the port that locks up is random. I've had both LAN and WAN lock up. I don't see any pattern to which port locks.

                        1 Reply Last reply Reply Quote 0
                        • M
                          mer
                          last edited by

                          It may be a good datapoint for cmb,  I'd suggest maybe getting the status and emailing it or a link as he asked another to do here.  That'll give 2 datapoints.

                          https://forum.pfsense.org/index.php?topic=110716.msg617252#msg617252

                          1 Reply Last reply Reply Quote 0
                          • S
                            SoloIT
                            last edited by

                            I have emailed cmb the status file.

                            1 Reply Last reply Reply Quote 0
                            • S
                              SoloIT
                              last edited by

                              We've locked up 2 more times this morning under fairly heavy IPSec VPN load.

                              1 Reply Last reply Reply Quote 0
                              • S
                                SoloIT
                                last edited by

                                Had another lockup with something new in the system log. Among the Watchdog timeout: kernel: sonewconn: pcb 0xfffff80016aaa960: Listen queue overflow: 193 already in queue awaiting acceptance (1 occurrences)

                                
Apr 27 13:08:21 wttlfirewall check_reload_status: Linkup starting em2
Apr 27 13:08:22 wttlfirewall check_reload_status: Linkup starting em2
Apr 27 13:08:22 wttlfirewall kernel: em2: link state changed to UP
Apr 27 13:09:04 wttlfirewall kernel: em2: Watchdog timeout -- resetting
Apr 27 13:09:04 wttlfirewall kernel: em2: link state changed to DOWN
Apr 27 13:09:04 wttlfirewall check_reload_status: Linkup starting em2
Apr 27 13:09:05 wttlfirewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Apr 27 13:09:05 wttlfirewall kernel: sonewconn: pcb 0xfffff80016aaa960: Listen queue overflow: 193 already in queue awaiting acceptance (1 occurrences)
Apr 27 13:09:05 wttlfirewall check_reload_status: Linkup starting em2
Apr 27 13:09:05 wttlfirewall kernel: em2: link state changed to UP
Apr 27 13:09:06 wttlfirewall check_reload_status: Could not connect to /var/run/php-fpm.socket

                                Not sure if that helps identify the issue or not

                                1 Reply Last reply Reply Quote 0
                                • M
                                  mer
                                  last edited by

                                  Well, it could be related, or maybe not.  What that error message is saying is something is trying to create a connection, but the application handling those connections is backed up.  It could be new connections are coming in faster than the app can handle or the app is not processing the connection (maybe something else running).

                                  Links to a couple of old threads that have good info:
                                  https://forums.freebsd.org/threads/43712/
                                  https://lists.freebsd.org/pipermail/freebsd-stable/2013-August/074561.html

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    afreaken
                                    last edited by

                                    Having the same issues as described in this thread. Using 2x pfSense Netgate SG-2440 systems. Both updated to 2.3, both experienced the lockup issue.

                                    They are using the igb nic's, 1 LAN, 1 WAN with an IPsec VPN between both locations.

                                    Last lockup happened at ~4am this morning which should not have much traffic at that time.

                                    System ran fairly well prior to upgrading from 2.2.6 - 2.3. Hopefully a fix is available soon, otherwise I'm going to have to revert back to 2.2.6.

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      mer
                                      last edited by

                                      @afreaken:

                                      Having the same issues as described in this thread. Using 2x pfSense Netgate SG-2440 systems. Both updated to 2.3, both experienced the lockup issue.

                                      They are using the igb nic's, 1 LAN, 1 WAN with an IPsec VPN between both locations.

                                      Last lockup happened at ~4am this morning which should not have much traffic at that time.

                                      System ran fairly well prior to upgrading from 2.2.6 - 2.3. Hopefully a fix is available soon, otherwise I'm going to have to revert back to 2.2.6.

                                      Interesting.  I think most/all folks having the same (or what appears to be the same) issue are all doing something with IPsec and em or igb interfaces

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        SoloIT
                                        last edited by

                                        I've started rolling all mine back to 2.2.6. Though I assume the team is working on the issue, I've heard nothing from them in 2 days. I've asked for a status update.

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          afreaken
                                          last edited by

                                          @SoloIT:

                                          I've started rolling all mine back to 2.2.6. Though I assume the team is working on the issue, I've heard nothing from them in 2 days. I've asked for a status update.

                                          know a safe source for 2.2.6? Don't think I downloaded it, just did the auto upgrade. Last version I have is 2.2.3, and I would rather get the last stable release (from a safe source).

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            SoloIT
                                            last edited by

                                            You can get them direct from pfSense: http://files.pfsense.org/mirror/downloads/old/

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post