Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    503 - Service Not Available and webintefrace really slow after upgrade

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    20 Posts 5 Posters 17.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W Offline
      wallabybob
      last edited by

      I think the RRD graph is showing a sudden jump in the number of processes (not CPU utilisation) from about 40 to about 100 at around 1230,

      The top output doesn't suggest your system is CPU bound - with four CPUs apparently idle there should be plenty of spare CPU.

      What about errors on the interfaces? What is reported by pfSense shell command # netstat -i

      1 Reply Last reply Reply Quote 0
      • T Offline
        thinktank
        last edited by

        Is this issue resolved? I'm having the exact same problem with my Dell R310. Tested with i386 and amd64 builds.

        1 Reply Last reply Reply Quote 0
        • W Offline
          wallabybob
          last edited by

          @thinktank:

          Is this issue resolved?

          Apparently not in that the information flow dried up leaving a couple of unanswered questions.

          @thinktank:

          I'm having the exact same problem with my Dell R310.

          Then please provide the information requested in earlier replies. And when providing the top -S -H output please don't cut off the top few lines which give the number of processes and memory and swap use summary.

          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            Anybody using Dell hardware with a multiport broadcom based NIC should at least try the broadcom settings here:
            http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
            This seems to cause a range of symptoms.

            Steve

            1 Reply Last reply Reply Quote 0
            • T Offline
              thinktank
              last edited by

              It seems I have narrowed my problem to the bcm network cards. Whenever I plug a network cable into one of these, the system becomes unresponsive.
              I can't even get a terminal session, so i can't run top command.

              I'll take stephenw10's advice and give the tweaking a shot. I'll post back with the results.

              1 Reply Last reply Reply Quote 0
              • T Offline
                thinktank
                last edited by

                I disabled the ports connected on my switch connected to the broadcom network cards and my system became responsive again. I then logged in to the box with ssh, started top -S -H and enabled the ports again. Voila the box became totally unresponsive again. Top command also seems to be locked up. Here is the output from the top command:

                
last pid: 17723;  load averages:  0.00,  0.00,  0.00                                                                                         up 0+19:29:52  10:45:26
143 processes: 9 running, 94 sleeping, 40 waiting
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 40M Active, 13M Inact, 115M Wired, 84K Cache, 24M Buf, 2820M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root     171 ki31     0K    64K CPU7    7  19.4H 100.00% {idle: cpu7}
   11 root     171 ki31     0K    64K CPU6    6  19.4H 100.00% {idle: cpu6}
   11 root     171 ki31     0K    64K CPU5    5  19.4H 100.00% {idle: cpu5}
   11 root     171 ki31     0K    64K CPU4    4  19.4H 100.00% {idle: cpu4}
   11 root     171 ki31     0K    64K CPU3    3  19.4H 100.00% {idle: cpu3}
   11 root     171 ki31     0K    64K CPU1    1  19.4H 100.00% {idle: cpu1}
   11 root     171 ki31     0K    64K RUN     2  19.4H 100.00% {idle: cpu2}
   11 root     171 ki31     0K    64K CPU0    0  19.4H 99.27% {idle: cpu0}
   12 root     -32    -     0K   320K WAIT    1   2:40  0.00% {swi4: clock}
    0 root      76    0     0K   176K sched   0   1:18  0.00% {swapper}
18965 root      44    0  3316K  1340K select  2   0:18  0.00% apinger
   12 root     -44    -     0K   320K WAIT    3   0:05  0.00% {swi1: netisr 0}
    3 root      -8    -     0K     8K -       0   0:02  0.00% g_up
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq20: atapci0}
   12 root     -68    -     0K   320K WAIT    0   0:02  0.00% {irq256: igb0:que}
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq22: ehci0 ehc}
   14 root     -16    -     0K     8K -       2   0:02  0.00% yarrow
28786 root      44    0 53596K 20084K lockf   5   0:01  0.00% php
   12 root     -32    -     0K   320K WAIT    1   0:01  0.00% {swi4: clock}
    2 root      -8    -     0K     8K -       2   0:01  0.00% g_event
   15 root     -64    -     0K    64K -       5   0:01  0.00% {usbus0}
   12 root     -68    -     0K   320K WAIT    7   0:01  0.00% {irq274: bce0}
    9 root      -8    -     0K     8K m:w1    5   0:01  0.00% g_mirror pfSenseMir
    4 root      -8    -     0K     8K -       0   0:01  0.00% g_down
   15 root     -64    -     0K    64K -       2   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    4   0:01  0.00% {swi4: clock}
   15 root     -64    -     0K    64K -       3   0:01  0.00% {usbus0}
   23 root      44    -     0K     8K syncer  2   0:01  0.00% syncer
   15 root     -64    -     0K    64K -       6   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    5   0:00  0.00% {irq261: igb0:que}
12474 root      44    0  5912K  2352K bpf     0   0:00  0.00% tcpdump
24959 root      57    0 54620K 20024K keglim  2   0:00  0.00% php
   12 root     -24    -     0K   320K WAIT    0   0:00  0.00% {swi6: task queue}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -32    -     0K   320K WAIT    1   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    1   0:00  0.00% {irq257: igb0:que}
   12 root     -68    -     0K   320K WAIT    3   0:00  0.00% {irq259: igb0:que}
25118 root      44    0 53596K 16780K accept  0   0:00  0.00% php
    8 root      44    -     0K     8K pftm    2   0:00  0.00% pfpurge
   24 root      59    -     0K     8K sdflus  2   0:00  0.00% softdepflush
   22 root      59    -     0K     8K vlruwt  2   0:00  0.00% vnlru
   12 root     -68    -     0K   320K WAIT    7   0:00  0.00% {irq263: igb0:que}
   21 root      59    -     0K     8K psleep  2   0:00  0.00% bufdaemon
13518 root      45    0 53596K 19708K keglim  5   0:00  0.00% php
   12 root     -32    -     0K   320K WAIT    3   0:00  0.00% {swi4: clock}
12091 root      44    0  4944K  2492K select  7   0:00  0.00% syslogd
   12 root     -32    -     0K   320K WAIT    7   0:00  0.00% {swi4: clock}
42301 root      44    0  3712K  2028K CPU2    2   0:00  0.00% top
25939 root      64   20  6588K  4336K keglim  6   0:00  0.00% lighttpd
42511 _ntp      44    0  3316K  1344K select  2   0:00  0.00% ntpd
   12 root     -68    -     0K   320K WAIT    6   0:00  0.00% {irq262: igb0:que}
   20 root      76 ki-6     0K     8K pollid  2   0:00  0.00% idlepoll
58160 root      44    0  3404K  1372K nanslp  2   0:00  0.00% cron
   40 root      -8    -     0K     8K mdwait  2   0:00  0.00% md0
 1389 root      64   20  7992K  3544K select  7   0:00  0.00% sshd
 6888 root      68    0  3316K  1040K nanslp  0   0:00  0.00% minicron
   17 root      59    -     0K     8K psleep  0   0:00  0.00% pagedaemon
                1 Reply Last reply Reply Quote 0
                • T Offline
                  thinktank
                  last edited by

                  After taking this textdump I had to reboot since the system became unresponsive. Currently the system is not booting properly. It hangs after Starting Cron … Done.
                  No debug messages. I haven't added the tuning parameters in loader.local yet, so i know thats not the problem.

                  I'm one inch from using the two R310's as paperweights and getting new hardware  ;D

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    This looks a lot like the broadcom problem mentioned in the guide. Something in the driver/hardware combination causes it to use way more mbufs than other cards. If you get it to reboot what is your mbuf usage on the dashboard?
                    Alternatively:

                    No problems on my boxes with that driver but if you are on the 64bit build do a netstat -m and check the following line

                    0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

                    If they are NOT zero then increase the value of kern.ipc.nmbclusters.

                    Are you running 64bit?

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • T Offline
                      thinktank
                      last edited by

                      At first I went for 64-bit, but after a while of headbashing i swapped it in for 32-bit.

                      Didn't manage to get the 32-bit booting again so now i'm back in 64-bit.

                      I'll do some checking today and i'll post all debug informasjon I can get hold of.

                      1 Reply Last reply Reply Quote 0
                      • T Offline
                        thinktank
                        last edited by

                        Here is a little update.

                        I have just reinstalled Pfsense with 64-bit. I'm currently only using the two Intel cards.

                        System seems responsive so far but the MBUF is: 17670/25600.

                        1 Reply Last reply Reply Quote 0
                        • T Offline
                          thinktank
                          last edited by

                          Added the following to loader.conf.local:

                          kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                          This seems to have fixed the issues. I'll let it run for week and do some stresstesting, but so far it seems stable.

                          A big thanks to wallabybob and stephenw10 for helping me out!

                          1 Reply Last reply Reply Quote 0
                          • S Offline
                            stratoscott
                            last edited by

                            Hello,
                            these settings are avalaible on pf 2.0.1 amd64 ?
                            Thanks

                            1 Reply Last reply Reply Quote 0
                            • T Offline
                              thinktank
                              last edited by

                              Hi,

                              Yes, you have to create the loader.conf.local in /boot/

                              Add these three lines:

                              kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                              And reboot.

                              1 Reply Last reply Reply Quote 0
                              • M Offline
                                MaxFontana
                                last edited by

                                Sorry for this long delay. Being actually the one who posted this request, implies at least an update from me ;-)
                                I can confirm that modifying the above mentioned files did the trick! Everything running flawlessly! ( using a del r210 with a Broadcom card and an Intel I3 cpu)
                                Thanks so much for your help!
                                Regards
                                max
                                Italy

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.