Realtek NIC + PPPoe + Gig, Cpu usage seems odd?
-
I am aware of all the issues with Realtek Nics and PPPoe and Gig speeds I have already applied the net.isr.dispatch="deferred" fix and running the compile 1.95 driver.
That gave me another 30% increase in speed. But I was curious is anyone can explain the output below from Netstat -Q.
From what i am reading it seems that it "favors" one Cpu(Cpu1) and never touches Cpu0 for ip??
Configuration: Setting Current Limit Thread count 4 4 Default queue limit 256 10240 Dispatch policy deferred n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 1000 flow default --- igmp 2 256 source default --- rtsock 3 1024 source default --- arp 4 256 source default --- ether 5 256 source direct --- ip6 6 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 0 0 0 0 0 0 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 2 0 0 0 416 416 0 0 arp 0 0 0 0 0 0 0 0 0 ether 0 0 2283875 0 0 0 2283875 0 0 ip6 0 0 0 0 0 0 0 1 1 ip 0 1000 0 0 7907 3403871 3403871 1 1 igmp 0 0 0 0 0 0 0 1 1 rtsock 0 0 0 0 0 0 0 1 1 arp 0 1 0 0 0 46 46 1 1 ether 0 0 2922613 0 0 0 2922613 1 1 ip6 0 1 0 0 0 6 6 2 2 ip 0 3 0 0 0 9386 9386 2 2 igmp 0 0 0 0 0 0 0 2 2 rtsock 0 0 0 0 0 0 0 2 2 arp 0 0 0 0 0 0 0 2 2 ether 0 0 2865 0 0 0 2865 2 2 ip6 0 0 0 0 0 0 0 3 3 ip 0 2 0 0 0 1305 1305 3 3 igmp 0 0 0 0 0 0 0 3 3 rtsock 0 0 0 0 0 0 0 3 3 arp 0 1 0 0 0 16 16 3 3 ether 0 0 2902 0 0 0 2902 3 3 ip6 0 1 0 0 0 2 2Ran a few more tests
net.isr.maxthreads="3" net.isr.direct="0" net.isr.direct_force="0"Has netted me at least 2 cpu's working now, forcing it to 3 (4 total) caused it to use it more evenly? Setting it to 4 showed no gains.
Configuration: Setting Current Limit Thread count 3 3 Default queue limit 256 10240 Dispatch policy deferred n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 1000 flow default --- igmp 2 256 source default --- rtsock 3 1024 source default --- arp 4 256 source default --- ether 5 256 source direct --- ip6 6 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 901 0 0 0 2859408 2859408 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 2 0 0 0 546 546 0 0 arp 0 0 0 0 0 0 0 0 0 ether 0 0 3307416 0 0 0 3307416 0 0 ip6 0 1 0 0 0 15 15 1 1 ip 0 1000 0 0 4381 2817582 2817582 1 1 igmp 0 0 0 0 0 0 0 1 1 rtsock 0 0 0 0 0 0 0 1 1 arp 0 1 0 0 0 1230 1230 1 1 ether 0 0 5272237 0 0 0 5272237 1 1 ip6 0 1 0 0 0 7 7 2 2 ip 0 0 0 0 0 0 0 2 2 igmp 0 0 0 0 0 0 0 2 2 rtsock 0 0 0 0 0 0 0 2 2 arp 0 0 0 0 0 0 0 2 2 ether 0 0 5874 0 0 0 5874 2 2 ip6 0 0 0 0 0 0 0Still lots of Q drops. but will see if i can fix those.
Running at ~ 800/750 Mbps now.
-
That seems expected for PPPoE. Only one CPU core is used.
Unless you're indicating something deeper?
Steve
-
@stephenw10 said in Realtek NIC + PPPoe + Gig, Cpu usage seems odd?:
That seems expected for PPPoE. Only one CPU core is used.
Unless you're indicating something deeper?
Steve
Well by making the tweak I was able to to get CPU0 working along with CPU1, which netted me another 10-15% worth of performance. PPPoe only using 1 core is very limiting in order to get the full speeds of my Gig line. Odd that I was able to get it to run with 2 cores by specifying
net.isr.maxthreads="3" net.isr.direct="0" net.isr.direct_force="0"But i cannot seem to get it to use three. Also any ideas on what i can do for the QDrops?
-
Mmm, I'd have to read into it to answer that in any depth but as I understand it PPPoE is fundamentally single threaded and changing that is non-trivial. No amount if changing settings is going to change that.
I would say what you;re seeing there is the limit of what the single core can do combined with the Realtek driver with one core in servicing the queues in each direction.
You are seeing queue drops because it can't service the queues fast enough. I would have to guess that's the upload queue.What CPU is that?
Steve
-
it's a Intel(R) Celeron(R) CPU N3150 @ 1.60GHz. I see activity on both CPU's (Qdrops, Queued and Handled) when running a download/upload test but only see Len packets on the ip service of CPU0 when downloading and CPU1 when uploading. Is there a ressourrce i can look up to go into more details as to what all the headers are? Len, Wmark.. and so on.
Sample
Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 1000 0 0 15609 193073039 193073039 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 3 0 0 0 4725 4725 0 0 arp 0 0 0 0 0 0 0 0 0 ether 0 0 189534247 0 0 0 189534247 0 0 ip6 0 2 0 0 0 8324 8324 1 1 ip 674 1000 0 0 57090 120420852 120419444 1 1 igmp 0 0 0 0 0 0 0 1 1 rtsock 0 0 0 0 0 0 0 1 1 arp 0 2 0 0 0 44889 44889 1 1 ether 0 0 310688209 0 0 0 310688209 1 1 ip6 0 2 0 0 0 12014 12014 2 2 ip 0 0 0 0 0 0 0 2 2 igmp 0 0 0 0 0 0 0 2 2 rtsock 0 0 0 0 0 0 0 2 2 arp 0 0 0 0 0 0 0 2 2 ether 0 0 1016577 0 0 0 1016577 2 2 ip6 0 0 0 0 0 0 0 -
I don't know of anything specific for that display but there are a number of FreeBSD man pages related to net isr.
I think this may be confusing matters though, PPPoE is not IP. I suspect the deferred setting there allows netgraph to work better. Though that's a guess.
The basic issue there though is that your CPU is not fast enough so the queues are filling and dropping packets once they are full. I have assumed the Len column to be the queue length at that instant and WMark to be the high water mark. In your output the queue that hits a watermark of 1000 has dropped packets as expected.
To compare here's output from my box I was testing PPPoE throughput on:
[2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: netstat -Q Configuration: Setting Current Limit Thread count 4 4 Default queue limit 256 10240 Dispatch policy deferred n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 1000 flow default --- igmp 2 256 source default --- rtsock 3 1024 source default --- arp 4 256 source default --- ether 5 256 source direct --- ip6 6 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 54 477871 0 0 12536027 13013897 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 2 0 0 0 36549 36549 0 0 arp 0 0 342 0 0 0 342 0 0 ether 0 0 752091 0 0 0 752091 0 0 ip6 0 0 2011 0 0 0 2011 1 1 ip 0 11 536285 0 0 210576 746861 1 1 igmp 0 0 0 0 0 0 0 1 1 rtsock 0 0 0 0 0 0 0 1 1 arp 0 1 0 0 0 2384 2384 1 1 ether 0 0 511938 0 0 0 511938 1 1 ip6 0 2 0 0 0 50151 50151 2 2 ip 0 1 7995646 0 0 184 7995830 2 2 igmp 0 0 0 0 0 0 0 2 2 rtsock 0 0 0 0 0 0 0 2 2 arp 0 0 0 0 0 0 0 2 2 ether 0 0 19436994 0 0 0 19436994 2 2 ip6 0 0 0 0 0 0 0 3 3 ip 0 1 1948445 0 0 160 1948605 3 3 igmp 0 0 0 0 0 0 0 3 3 rtsock 0 0 0 0 0 0 0 3 3 arp 0 0 0 0 0 0 0 3 3 ether 0 0 1948571 0 0 0 1948571 3 3 ip6 0 0 0 0 0 0 0The queues hardly get full at all and as a result no packets are dropped.
Do you have powerd enabled? Is the CPU able to use 'burst' mode?
Steve
-
Thanks for the Powerd clue, i guess the cpu was not "bursting" to 2.08Ghz without it on, not sure yet but it has gained me another 100Mbps , closing in on 900 now..
