Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    503 - Service Not Available and webintefrace really slow after upgrade

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    20 Posts 5 Posters 17.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T Offline
      thinktank
      last edited by

      It seems I have narrowed my problem to the bcm network cards. Whenever I plug a network cable into one of these, the system becomes unresponsive.
      I can't even get a terminal session, so i can't run top command.

      I'll take stephenw10's advice and give the tweaking a shot. I'll post back with the results.

      1 Reply Last reply Reply Quote 0
      • T Offline
        thinktank
        last edited by

        I disabled the ports connected on my switch connected to the broadcom network cards and my system became responsive again. I then logged in to the box with ssh, started top -S -H and enabled the ports again. Voila the box became totally unresponsive again. Top command also seems to be locked up. Here is the output from the top command:

        
last pid: 17723;  load averages:  0.00,  0.00,  0.00                                                                                         up 0+19:29:52  10:45:26
143 processes: 9 running, 94 sleeping, 40 waiting
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 40M Active, 13M Inact, 115M Wired, 84K Cache, 24M Buf, 2820M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root     171 ki31     0K    64K CPU7    7  19.4H 100.00% {idle: cpu7}
   11 root     171 ki31     0K    64K CPU6    6  19.4H 100.00% {idle: cpu6}
   11 root     171 ki31     0K    64K CPU5    5  19.4H 100.00% {idle: cpu5}
   11 root     171 ki31     0K    64K CPU4    4  19.4H 100.00% {idle: cpu4}
   11 root     171 ki31     0K    64K CPU3    3  19.4H 100.00% {idle: cpu3}
   11 root     171 ki31     0K    64K CPU1    1  19.4H 100.00% {idle: cpu1}
   11 root     171 ki31     0K    64K RUN     2  19.4H 100.00% {idle: cpu2}
   11 root     171 ki31     0K    64K CPU0    0  19.4H 99.27% {idle: cpu0}
   12 root     -32    -     0K   320K WAIT    1   2:40  0.00% {swi4: clock}
    0 root      76    0     0K   176K sched   0   1:18  0.00% {swapper}
18965 root      44    0  3316K  1340K select  2   0:18  0.00% apinger
   12 root     -44    -     0K   320K WAIT    3   0:05  0.00% {swi1: netisr 0}
    3 root      -8    -     0K     8K -       0   0:02  0.00% g_up
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq20: atapci0}
   12 root     -68    -     0K   320K WAIT    0   0:02  0.00% {irq256: igb0:que}
   12 root     -64    -     0K   320K WAIT    7   0:02  0.00% {irq22: ehci0 ehc}
   14 root     -16    -     0K     8K -       2   0:02  0.00% yarrow
28786 root      44    0 53596K 20084K lockf   5   0:01  0.00% php
   12 root     -32    -     0K   320K WAIT    1   0:01  0.00% {swi4: clock}
    2 root      -8    -     0K     8K -       2   0:01  0.00% g_event
   15 root     -64    -     0K    64K -       5   0:01  0.00% {usbus0}
   12 root     -68    -     0K   320K WAIT    7   0:01  0.00% {irq274: bce0}
    9 root      -8    -     0K     8K m:w1    5   0:01  0.00% g_mirror pfSenseMir
    4 root      -8    -     0K     8K -       0   0:01  0.00% g_down
   15 root     -64    -     0K    64K -       2   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    4   0:01  0.00% {swi4: clock}
   15 root     -64    -     0K    64K -       3   0:01  0.00% {usbus0}
   23 root      44    -     0K     8K syncer  2   0:01  0.00% syncer
   15 root     -64    -     0K    64K -       6   0:01  0.00% {usbus1}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    5   0:00  0.00% {irq261: igb0:que}
12474 root      44    0  5912K  2352K bpf     0   0:00  0.00% tcpdump
24959 root      57    0 54620K 20024K keglim  2   0:00  0.00% php
   12 root     -24    -     0K   320K WAIT    0   0:00  0.00% {swi6: task queue}
   12 root     -32    -     0K   320K WAIT    2   0:00  0.00% {swi4: clock}
   12 root     -32    -     0K   320K WAIT    1   0:00  0.00% {swi4: clock}
   12 root     -68    -     0K   320K WAIT    1   0:00  0.00% {irq257: igb0:que}
   12 root     -68    -     0K   320K WAIT    3   0:00  0.00% {irq259: igb0:que}
25118 root      44    0 53596K 16780K accept  0   0:00  0.00% php
    8 root      44    -     0K     8K pftm    2   0:00  0.00% pfpurge
   24 root      59    -     0K     8K sdflus  2   0:00  0.00% softdepflush
   22 root      59    -     0K     8K vlruwt  2   0:00  0.00% vnlru
   12 root     -68    -     0K   320K WAIT    7   0:00  0.00% {irq263: igb0:que}
   21 root      59    -     0K     8K psleep  2   0:00  0.00% bufdaemon
13518 root      45    0 53596K 19708K keglim  5   0:00  0.00% php
   12 root     -32    -     0K   320K WAIT    3   0:00  0.00% {swi4: clock}
12091 root      44    0  4944K  2492K select  7   0:00  0.00% syslogd
   12 root     -32    -     0K   320K WAIT    7   0:00  0.00% {swi4: clock}
42301 root      44    0  3712K  2028K CPU2    2   0:00  0.00% top
25939 root      64   20  6588K  4336K keglim  6   0:00  0.00% lighttpd
42511 _ntp      44    0  3316K  1344K select  2   0:00  0.00% ntpd
   12 root     -68    -     0K   320K WAIT    6   0:00  0.00% {irq262: igb0:que}
   20 root      76 ki-6     0K     8K pollid  2   0:00  0.00% idlepoll
58160 root      44    0  3404K  1372K nanslp  2   0:00  0.00% cron
   40 root      -8    -     0K     8K mdwait  2   0:00  0.00% md0
 1389 root      64   20  7992K  3544K select  7   0:00  0.00% sshd
 6888 root      68    0  3316K  1040K nanslp  0   0:00  0.00% minicron
   17 root      59    -     0K     8K psleep  0   0:00  0.00% pagedaemon
        1 Reply Last reply Reply Quote 0
        • T Offline
          thinktank
          last edited by

          After taking this textdump I had to reboot since the system became unresponsive. Currently the system is not booting properly. It hangs after Starting Cron … Done.
          No debug messages. I haven't added the tuning parameters in loader.local yet, so i know thats not the problem.

          I'm one inch from using the two R310's as paperweights and getting new hardware  ;D

          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            This looks a lot like the broadcom problem mentioned in the guide. Something in the driver/hardware combination causes it to use way more mbufs than other cards. If you get it to reboot what is your mbuf usage on the dashboard?
            Alternatively:

            No problems on my boxes with that driver but if you are on the 64bit build do a netstat -m and check the following line

            0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

            If they are NOT zero then increase the value of kern.ipc.nmbclusters.

            Are you running 64bit?

            Steve

            1 Reply Last reply Reply Quote 0
            • T Offline
              thinktank
              last edited by

              At first I went for 64-bit, but after a while of headbashing i swapped it in for 32-bit.

              Didn't manage to get the 32-bit booting again so now i'm back in 64-bit.

              I'll do some checking today and i'll post all debug informasjon I can get hold of.

              1 Reply Last reply Reply Quote 0
              • T Offline
                thinktank
                last edited by

                Here is a little update.

                I have just reinstalled Pfsense with 64-bit. I'm currently only using the two Intel cards.

                System seems responsive so far but the MBUF is: 17670/25600.

                1 Reply Last reply Reply Quote 0
                • T Offline
                  thinktank
                  last edited by

                  Added the following to loader.conf.local:

                  kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                  This seems to have fixed the issues. I'll let it run for week and do some stresstesting, but so far it seems stable.

                  A big thanks to wallabybob and stephenw10 for helping me out!

                  1 Reply Last reply Reply Quote 0
                  • S Offline
                    stratoscott
                    last edited by

                    Hello,
                    these settings are avalaible on pf 2.0.1 amd64 ?
                    Thanks

                    1 Reply Last reply Reply Quote 0
                    • T Offline
                      thinktank
                      last edited by

                      Hi,

                      Yes, you have to create the loader.conf.local in /boot/

                      Add these three lines:

                      kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

                      And reboot.

                      1 Reply Last reply Reply Quote 0
                      • M Offline
                        MaxFontana
                        last edited by

                        Sorry for this long delay. Being actually the one who posted this request, implies at least an update from me ;-)
                        I can confirm that modifying the above mentioned files did the trick! Everything running flawlessly! ( using a del r210 with a Broadcom card and an Intel I3 cpu)
                        Thanks so much for your help!
                        Regards
                        max
                        Italy

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.