Multicore doesn't seem to be working?



  • I am using 2.2.2 nanobsd with a Celeron J1900, specifically a http://www.supermicro.com/products/motherboard/celeron/X10/X10SBA-L.cfm, and it doesn't seem to be using all the cores properly. I am only getting around 45MB/s over CIFS and top has a single core maxed and the interrupts doesn't look that bad. Here is the output during a transfer, the [kernel] jumps up to high 90's then over 20 seconds or so maxes out at the slightly above 100% as shown:

    last pid: 66260;  load averages:  2.41,  1.01,  0.46                                            up 0+00:06:58  20:43:31
    60 processes:  2 running, 57 sleeping, 1 waiting
    CPU 0:  0.0% user,  0.0% nice,  0.0% system, 18.3% interrupt, 81.7% idle
    CPU 1:  0.0% user,  0.0% nice,  1.1% system,  0.4% interrupt, 98.5% idle
    CPU 2:  0.0% user,  0.0% nice,  0.4% system,  1.5% interrupt, 98.1% idle
    CPU 3:  0.0% user,  0.0% nice, 99.6% system,  0.0% interrupt,  0.4% idle
    Mem: 63M Active, 33M Inact, 180M Wired, 1024K Cache, 66M Buf, 7498M Free
    Swap:
    
      PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
       11 root        4 155 ki31     0K    64K RUN     3  25:01 282.03% [idle]
        0 root       22  -8    0     0K   352K -       1   1:44 100.78% [kernel]
       12 root       28 -72    -     0K   448K WAIT    3   0:43  17.82% [intr]
        5 root        1 -16    -     0K    16K -       1   0:02   0.00% [fdc0]
    20944 root        1  26    0   228M 42404K accept  1   0:00   0.00% php-fpm: pool lighty (php-fpm)
       16 root        4 -68    -     0K    64K -       1   0:00   0.00% [usb]
    30837 unbound     4  20    0 59572K 19228K kqread  1   0:00   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.con
    35072 root        1  20    0 21996K  3160K CPU1    1   0:00   0.00% top -aSP
       15 root        1 -16    -     0K    16K -       1   0:00   0.00% [rand_harvestq]
    72775 root        1  20    0 28176K 18060K select  1   0:00   0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /v
       57 root        1  -8    -     0K    16K mdwait  0   0:00   0.00% [md1]
        6 root        1 -16    -     0K    16K pftm    1   0:00   0.00% [pf purge]
        4 root        2 -16    -     0K    32K -       0   0:00   0.00% [cam]
    21173 root        1  20    0 12464K  2176K select  2   0:00   0.00% /usr/local/sbin/apinger -c /var/etc/apinger.conf
    16793 root        1  20    0 16812K  2320K bpf     1   0:00   0.00% /usr/local/sbin/filterlog -i pflog0 -p /var/run/fil
        1 root        1  20    0  9472K   852K wait    1   0:00   0.00% [init]
    47353 root        1  52   20 17144K  2536K wait    3   0:00   0.00% /bin/sh /var/db/rrd/updaterrd.sh
    31990 root        1  20    0 55636K  6056K select  0   0:00   0.00% sshd: root@pts/0 (sshd)
    62506 root        1  20    0 14664K  2416K select  2   0:00   0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/lo
       14 root        3  -8    -     0K    48K -       0   0:00   0.00% [geom]
    36625 dhcpd       1  20    0 24820K 12516K select  1   0:00   0.00% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chr
      283 root        1  20    0   224M 24208K kqread  1   0:00   0.00% php-fpm: master process (/usr/local/lib/php-fpm.con
    34471 root        1  20    0 17484K  3456K pause   3   0:00   0.00% /bin/tcsh
    25347 root        1  20    0 54896K  5528K kqread  1   0:00   0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-webConf
    69980 root        1  52    0 43576K  2700K wait    2   0:00   0.00% login [pam] (login)
    
    

    This is a pretty clean install, I have only added a few NATs. Currently I have TSO and LRO disabled and chucksum enabled. I have tried toggling them in various ways, but it doesn't help. I have increased nmbclusters to 1M, tried hw.igb.fc_setting=0 and net.inet.tcp.tso=0 and none help.



  • A bit of Googling says CIFS/SAMBA may not support threading. https://www.samba.org/samba/docs/man/Samba-Developers-Guide/architecture.html

    Are you sure CIFS can make use of SMP?



  • I'm pretty sure CIFS isn't the issue, I was getting 110MBps over my 1Gb connection and now get 225MBps reads and 800MBps writes over my 10Gb connection.



  • The firewall cannot use SMP to accelerate a single data flow, but does so to accelerate many flows. CIFS is probably only using a single flow which is why only one core is getting used.

    Why that single flow is running a single core is the bottleneck. I wonder what the heck "kernel" is that it's using 100% of a core. 17% interrupts is pretty high, but that may be normal for a 10watt CPU.


  • Banned

    Please, stop using CIFS to test. You won't get meaningful results. For SMB to use multiple TCP connections, you need SMB 3.0 with multichannel on both sides, with Receive Side Scaling (RSS) enabled NIC.



  • Ah I see what you mean now, then everything seems fine except the interrupts being a bit higher than you think they should be. Thanks I will see if I can get them a bit lower.