Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    503 - Service Not Available and webintefrace really slow after upgrade

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    20 Posts 5 Posters 17.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S Offline
      stephenw10 Netgate Administrator
      last edited by

      Anybody using Dell hardware with a multiport broadcom based NIC should at least try the broadcom settings here:
      http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
      This seems to cause a range of symptoms.

      Steve

      1 Reply Last reply Reply Quote 0
      • T Offline
        thinktank
        last edited by

        It seems I have narrowed my problem to the bcm network cards. Whenever I plug a network cable into one of these, the system becomes unresponsive.
        I can't even get a terminal session, so i can't run top command.

        I'll take stephenw10's advice and give the tweaking a shot. I'll post back with the results.

        1 Reply Last reply Reply Quote 0
        • T Offline
          thinktank
          last edited by

          I disabled the ports connected on my switch connected to the broadcom network cards and my system became responsive again. I then logged in to the box with ssh, started top -S -H and enabled the ports again. Voila the box became totally unresponsive again. Top command also seems to be locked up. Here is the output from the top command:

          
last pid: 17723;  load averages:  0.00,  0.00,  0.00                                                                                         up 0+19:29:52  10:45:26
143 processes: 9 running, 94 sleeping, 40 waiting
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 40M Active, 13M Inact, 115M Wired, 84K Cache, 24M Buf, 2820M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root     171 ki31     0K    64K CPU7    7  19.4H 100.00% {idle: cpu7}
   11 root     171 ki31     0K    64K CPU6    6  19.4H 100.00% {idle: cpu6}
   11 root     171 ki31     0K    64K CPU5    5  19.4H 100.00% {idle: cpu5}
   11 root     171 ki31     0K    64K CPU4    4  19.4H 100.00% {idle: cpu4}
   11 root     171 ki31     0K    64K CPU3    3  19.4H 100.00% {idle: cpu3}
   11 root     171 ki31     0K    64K CPU1    1  19.4H 100.00% {idle: cpu1}
   11 root     171 ki31     0K    64K RUN     2  19.4H 100.00% {idle: cpu2}
   11 root     171 ki31     0K    64K CPU0    0  19.4H 99.27% {idle: cpu0}
   12 root     -32    -     0K   320K WAIT    1   2:40  0.00% {swi4: clock}
    0 root      76    0     0K   176K sched   0   1:18  0.00% {swapper}
18965 root      44    0  3316K  1340K select  2   0:18  0.00% apinger
   12 root     -44    -     0K   320K WAIT    3   0:05  0.00% {swi1: netisr 0}
    3 root      -8    -     0K     8K -       0   0:02  0.00% g_up
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq20: atapci0}
   12 root     -68    -     0K   320K WAIT    0   0:02  0.00% {irq256: igb0:que}
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq22: ehci0 ehc}
   14 root     -16    -     0K     8K -       2   0:02  0.00% yarrow
28786 root      44    0 53596K 20084K lockf   5   0:01  0.00% php
   12 root     -32    -     0K   320K WAIT    1   0:01  0.00% {swi4: clock}
    2 root      -8    -     0K     8K -       2   0:01  0.00% g_event
   15 root     -64    -     0K    64K -       5   0:01  0.00% {usbus0}
   12 root     -68    -     0K   320K WAIT    7   0:01  0.00% {irq274: bce0}
    9 root      -8    -     0K     8K m:w1    5   0:01  0.00% g_mirror pfSenseMir
    4 root      -8    -     0K     8K -       0   0:01  0.00% g_down
   15 root     -64    -     0K    64K -       2   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    4   0:01  0.00% {swi4: clock}
   15 root     -64    -     0K    64K -       3   0:01  0.00% {usbus0}
   23 root      44    -     0K     8K syncer  2   0:01  0.00% syncer
   15 root     -64    -     0K    64K -       6   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    5   0:00  0.00% {irq261: igb0:que}
12474 root      44    0  5912K  2352K bpf     0   0:00  0.00% tcpdump
24959 root      57    0 54620K 20024K keglim  2   0:00  0.00% php
   12 root     -24    -     0K   320K WAIT    0   0:00  0.00% {swi6: task queue}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -32    -     0K   320K WAIT    1   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    1   0:00  0.00% {irq257: igb0:que}
   12 root     -68    -     0K   320K WAIT    3   0:00  0.00% {irq259: igb0:que}
25118 root      44    0 53596K 16780K accept  0   0:00  0.00% php
    8 root      44    -     0K     8K pftm    2   0:00  0.00% pfpurge
   24 root      59    -     0K     8K sdflus  2   0:00  0.00% softdepflush
   22 root      59    -     0K     8K vlruwt  2   0:00  0.00% vnlru
   12 root     -68    -     0K   320K WAIT    7   0:00  0.00% {irq263: igb0:que}
   21 root      59    -     0K     8K psleep  2   0:00  0.00% bufdaemon
13518 root      45    0 53596K 19708K keglim  5   0:00  0.00% php
   12 root     -32    -     0K   320K WAIT    3   0:00  0.00% {swi4: clock}
12091 root      44    0  4944K  2492K select  7   0:00  0.00% syslogd
   12 root     -32    -     0K   320K WAIT    7   0:00  0.00% {swi4: clock}
42301 root      44    0  3712K  2028K CPU2    2   0:00  0.00% top
25939 root      64   20  6588K  4336K keglim  6   0:00  0.00% lighttpd
42511 _ntp      44    0  3316K  1344K select  2   0:00  0.00% ntpd
   12 root     -68    -     0K   320K WAIT    6   0:00  0.00% {irq262: igb0:que}
   20 root      76 ki-6     0K     8K pollid  2   0:00  0.00% idlepoll
58160 root      44    0  3404K  1372K nanslp  2   0:00  0.00% cron
   40 root      -8    -     0K     8K mdwait  2   0:00  0.00% md0
 1389 root      64   20  7992K  3544K select  7   0:00  0.00% sshd
 6888 root      68    0  3316K  1040K nanslp  0   0:00  0.00% minicron
   17 root      59    -     0K     8K psleep  0   0:00  0.00% pagedaemon
          1 Reply Last reply Reply Quote 0
          • T Offline
            thinktank
            last edited by

            After taking this textdump I had to reboot since the system became unresponsive. Currently the system is not booting properly. It hangs after Starting Cron … Done.
            No debug messages. I haven't added the tuning parameters in loader.local yet, so i know thats not the problem.

            I'm one inch from using the two R310's as paperweights and getting new hardware  ;D

            1 Reply Last reply Reply Quote 0
            • stephenw10S Offline
              stephenw10 Netgate Administrator
              last edited by

              This looks a lot like the broadcom problem mentioned in the guide. Something in the driver/hardware combination causes it to use way more mbufs than other cards. If you get it to reboot what is your mbuf usage on the dashboard?
              Alternatively:

              No problems on my boxes with that driver but if you are on the 64bit build do a netstat -m and check the following line

              0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

              If they are NOT zero then increase the value of kern.ipc.nmbclusters.

              Are you running 64bit?

              Steve

              1 Reply Last reply Reply Quote 0
              • T Offline
                thinktank
                last edited by

                At first I went for 64-bit, but after a while of headbashing i swapped it in for 32-bit.

                Didn't manage to get the 32-bit booting again so now i'm back in 64-bit.

                I'll do some checking today and i'll post all debug informasjon I can get hold of.

                1 Reply Last reply Reply Quote 0
                • T Offline
                  thinktank
                  last edited by

                  Here is a little update.

                  I have just reinstalled Pfsense with 64-bit. I'm currently only using the two Intel cards.

                  System seems responsive so far but the MBUF is: 17670/25600.

                  1 Reply Last reply Reply Quote 0
                  • T Offline
                    thinktank
                    last edited by

                    Added the following to loader.conf.local:

                    kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                    This seems to have fixed the issues. I'll let it run for week and do some stresstesting, but so far it seems stable.

                    A big thanks to wallabybob and stephenw10 for helping me out!

                    1 Reply Last reply Reply Quote 0
                    • S Offline
                      stratoscott
                      last edited by

                      Hello,
                      these settings are avalaible on pf 2.0.1 amd64 ?
                      Thanks

                      1 Reply Last reply Reply Quote 0
                      • T Offline
                        thinktank
                        last edited by

                        Hi,

                        Yes, you have to create the loader.conf.local in /boot/

                        Add these three lines:

                        kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                        And reboot.

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          MaxFontana
                          last edited by

                          Sorry for this long delay. Being actually the one who posted this request, implies at least an update from me ;-)
                          I can confirm that modifying the above mentioned files did the trick! Everything running flawlessly! ( using a del r210 with a Broadcom card and an Intel I3 cpu)
                          Thanks so much for your help!
                          Regards
                          max
                          Italy

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.