Multicore doesn't seem to be working?
-
I am using 2.2.2 nanobsd with a Celeron J1900, specifically a http://www.supermicro.com/products/motherboard/celeron/X10/X10SBA-L.cfm, and it doesn't seem to be using all the cores properly. I am only getting around 45MB/s over CIFS and top has a single core maxed and the interrupts doesn't look that bad. Here is the output during a transfer, the [kernel] jumps up to high 90's then over 20 seconds or so maxes out at the slightly above 100% as shown:
last pid: 66260; load averages: 2.41, 1.01, 0.46 up 0+00:06:58 20:43:31 60 processes: 2 running, 57 sleeping, 1 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 18.3% interrupt, 81.7% idle CPU 1: 0.0% user, 0.0% nice, 1.1% system, 0.4% interrupt, 98.5% idle CPU 2: 0.0% user, 0.0% nice, 0.4% system, 1.5% interrupt, 98.1% idle CPU 3: 0.0% user, 0.0% nice, 99.6% system, 0.0% interrupt, 0.4% idle Mem: 63M Active, 33M Inact, 180M Wired, 1024K Cache, 66M Buf, 7498M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 4 155 ki31 0K 64K RUN 3 25:01 282.03% [idle] 0 root 22 -8 0 0K 352K - 1 1:44 100.78% [kernel] 12 root 28 -72 - 0K 448K WAIT 3 0:43 17.82% [intr] 5 root 1 -16 - 0K 16K - 1 0:02 0.00% [fdc0] 20944 root 1 26 0 228M 42404K accept 1 0:00 0.00% php-fpm: pool lighty (php-fpm) 16 root 4 -68 - 0K 64K - 1 0:00 0.00% [usb] 30837 unbound 4 20 0 59572K 19228K kqread 1 0:00 0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.con 35072 root 1 20 0 21996K 3160K CPU1 1 0:00 0.00% top -aSP 15 root 1 -16 - 0K 16K - 1 0:00 0.00% [rand_harvestq] 72775 root 1 20 0 28176K 18060K select 1 0:00 0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /v 57 root 1 -8 - 0K 16K mdwait 0 0:00 0.00% [md1] 6 root 1 -16 - 0K 16K pftm 1 0:00 0.00% [pf purge] 4 root 2 -16 - 0K 32K - 0 0:00 0.00% [cam] 21173 root 1 20 0 12464K 2176K select 2 0:00 0.00% /usr/local/sbin/apinger -c /var/etc/apinger.conf 16793 root 1 20 0 16812K 2320K bpf 1 0:00 0.00% /usr/local/sbin/filterlog -i pflog0 -p /var/run/fil 1 root 1 20 0 9472K 852K wait 1 0:00 0.00% [init] 47353 root 1 52 20 17144K 2536K wait 3 0:00 0.00% /bin/sh /var/db/rrd/updaterrd.sh 31990 root 1 20 0 55636K 6056K select 0 0:00 0.00% sshd: root@pts/0 (sshd) 62506 root 1 20 0 14664K 2416K select 2 0:00 0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/lo 14 root 3 -8 - 0K 48K - 0 0:00 0.00% [geom] 36625 dhcpd 1 20 0 24820K 12516K select 1 0:00 0.00% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chr 283 root 1 20 0 224M 24208K kqread 1 0:00 0.00% php-fpm: master process (/usr/local/lib/php-fpm.con 34471 root 1 20 0 17484K 3456K pause 3 0:00 0.00% /bin/tcsh 25347 root 1 20 0 54896K 5528K kqread 1 0:00 0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-webConf 69980 root 1 52 0 43576K 2700K wait 2 0:00 0.00% login [pam] (login)This is a pretty clean install, I have only added a few NATs. Currently I have TSO and LRO disabled and chucksum enabled. I have tried toggling them in various ways, but it doesn't help. I have increased nmbclusters to 1M, tried hw.igb.fc_setting=0 and net.inet.tcp.tso=0 and none help.
-
A bit of Googling says CIFS/SAMBA may not support threading. https://www.samba.org/samba/docs/man/Samba-Developers-Guide/architecture.html
Are you sure CIFS can make use of SMP?
-
I'm pretty sure CIFS isn't the issue, I was getting 110MBps over my 1Gb connection and now get 225MBps reads and 800MBps writes over my 10Gb connection.
-
The firewall cannot use SMP to accelerate a single data flow, but does so to accelerate many flows. CIFS is probably only using a single flow which is why only one core is getting used.
Why that single flow is running a single core is the bottleneck. I wonder what the heck "kernel" is that it's using 100% of a core. 17% interrupts is pretty high, but that may be normal for a 10watt CPU.
-
Please, stop using CIFS to test. You won't get meaningful results. For SMB to use multiple TCP connections, you need SMB 3.0 with multichannel on both sides, with Receive Side Scaling (RSS) enabled NIC.
-
Ah I see what you mean now, then everything seems fine except the interrupts being a bit higher than you think they should be. Thanks I will see if I can get them a bit lower.