Network performance issue
-
I'm having some network performance issue with 1.2.3 RC3.
Running iperf only gives me about 500mb/s receive and 700mb/s send.
The system is an old P4 server board, with one onboard Intel gigabit nic attached directly to northbridge. The nic is capable of pushing over 940mb/s. I also have 2 dual port Intel MT 1000 nics attached to PCI-X slots. All the nics are getting the above performance. There was an irq sharing issue but I fixed it, still same results.
The destination system is a dual opteron server with 4 server nics (Intel MT dual port and Broadcom 5704 attached to seperate PCI-X bus). Testing between the destination server and other servers, I always get around over 940mb/s. I have a few FreeBSD 7.2 and 8.0 VMs, they all can get over 940mb/s even inside VMs.
I initially installed with UP kernel but performance was even worse, then I reloaded with SMP kernel, that's what the performance I got.
Any ideas? Thanks
-
UDP or TCP?
What if you disable packet filtering? -
UDP or TCP?
What if you disable packet filtering?TCP
PF rules for this internal network is set to allow everything now. still same.
-
can you post top -S ?
-
last pid: 47571; load averages: 0.00, 0.00, 0.00 up 0+18:36:34 19:00:26
89 processes: 2 running, 74 sleeping, 13 waiting
CPU: 0.0% user, 0.4% nice, 0.8% system, 0.4% interrupt, 98.5% idle
Mem: 57M Active, 22M Inact, 32M Wired, 16K Cache, 19M Buf, 377M Free
Swap: 1024M Total, 1024M FreePID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
11 root 1 171 ki31 0K 8K RUN 18.3H 100.00% idle: cpu0
28 root 1 -68 - 0K 8K WAIT 1:46 0.00% irq21: fxp0
25 root 1 -68 - 0K 8K - 1:03 0.00% em0 taskq
13 root 1 -32 - 0K 8K WAIT 1:00 0.00% swi4: clock sio
15 root 1 44 - 0K 8K - 0:18 0.00% yarrow
26 root 1 -68 - 0K 8K - 0:16 0.00% em1 taskq
614 root 1 4 0 5132K 3084K kqread 0:09 0.00% lighttpd
27 root 1 -68 - 0K 8K - 0:08 0.00% em2 taskq
361 root 1 8 0 3492K 1344K wait 0:07 0.00% sh
237 root 1 44 0 3268K 1152K select 0:06 0.00% syslogd
59242 root 1 4 0 40712K 14804K accept 0:06 0.00% php
31 root 1 0 - 0K 8K tzpoll 0:05 0.00% acpi_thermal
4 root 1 -8 - 0K 8K - 0:05 0.00% g_up
1181 root 1 8 20 3156K 800K nanslp 0:04 0.00% check_reload_status
368 root 1 -58 0 5716K 2836K bpf 0:04 0.00% tcpdump
5 root 1 -8 - 0K 8K - 0:03 0.00% g_down
369 root 1 -8 0 3156K 784K piperd 0:03 0.00% logger
4305 root 1 8 20 3492K 1492K wait 0:03 0.00% sh
658 root 1 4 0 43784K 18392K accept 0:03 0.00% php
2629 nobody 1 44 20 3156K 1304K select 0:02 0.00% dnsmasq
44 root 1 20 - 0K 8K syncer 0:02 0.00% syncer
12 root 1 -44 - 0K 8K WAIT 0:02 0.00% swi1: net
33 root 1 -8 - 0K 8K - 0:02 0.00% fdc0
3 root 1 -8 - 0K 8K - 0:02 0.00% g_event
22 root 1 8 - 0K 8K - 0:01 0.00% thread taskq
29 root 1 -64 - 0K 8K WAIT 0:01 0.00% irq14: ata0
639 root 1 4 0 45832K 19088K accept 0:01 0.00% php
618 root 1 4 0 45832K 19088K accept 0:01 0.00% php
37 root 1 8 - 0K 8K pftm 0:01 0.00% pfpurge
804 root 1 44 0 7012K 5880K select 0:01 0.00% bsnmpd
51 root 1 -8 - 0K 8K mdwait 0:01 0.00% md0
1255 root 1 44 0 7780K 3300K select 0:00 0.00% sshd
1141 root 1 44 0 3188K 1116K select 0:00 0.00% miniupnpd
45 root 1 -16 - 0K 8K sdflus 0:00 0.00% softdepflush
43 root 1 -4 - 0K 8K vlruwt 0:00 0.00% vnlru
42 root 1 -16 - 0K 8K psleep 0:00 0.00% bufdaemon
1156 root 1 8 0 3240K 1272K nanslp 0:00 0.00% cron
0 root 1 -16 0 0K 0K sched 0:00 0.00% swapper
2453 _dhcp 1 44 20 3156K 1316K select 0:00 0.00% dhclient
21 root 1 -28 - 0K 8K WAIT 0:00 0.00% swi5: +
41 root 1 171 ki31 0K 8K pollid 0:00 0.00% idlepoll
1183 root 1 8 0 3156K 900K nanslp 0:00 0.00% minicron
1286 root 1 20 0 4532K 2452K pause 0:00 0.00% tcsh
38 root 1 -16 - 0K 8K psleep 0:00 0.00% pagedaemon
2656 dhcpd 1 44 20 3156K 2028K select 0:00 0.00% dhcpdThanks for the response.
-
No, it is impossible.
I mean top -S on the firewall through which you are doing traffic test. -
last pid: 57439; load averages: 0.48, 0.16, 0.05 up 0+19:40:41 20:04:33
95 processes: 3 running, 79 sleeping, 13 waiting
CPU: 0.0% user, 0.0% nice, 99.3% system, 0.0% interrupt, 0.7% idle
Mem: 58M Active, 22M Inact, 33M Wired, 16K Cache, 19M Buf, 375M Free
Swap: 1024M Total, 1024M FreePID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
26 root 1 -68 - 0K 8K - 0:39 66.06% em1 taskq
11 root 1 171 ki31 0K 8K RUN 19.4H 17.09% idle: cpu0
57287 root 4 99 0 4864K 2024K RUN 0:03 15.19% iperf
28 root 1 -68 - 0K 8K WAIT 1:47 0.00% irq21: fxp0
13 root 1 -32 - 0K 8K WAIT 1:04 0.00% swi4: clock sio
25 root 1 -68 - 0K 8K - 1:03 0.00% em0 taskq
15 root 1 44 - 0K 8K - 0:18 0.00% yarrow
614 root 1 4 0 5132K 3084K kqread 0:09 0.00% lighttpd
27 root 1 -68 - 0K 8K - 0:08 0.00% em2 taskq
361 root 1 8 0 3492K 1344K wait 0:07 0.00% sh
237 root 1 44 0 3268K 1152K select 0:07 0.00% syslogd
59242 root 1 4 0 40712K 14804K accept 0:06 0.00% php
31 root 1 0 - 0K 8K tzpoll 0:05 0.00% acpi_thermal
4 root 1 -8 - 0K 8K - 0:05 0.00% g_up
1181 root 1 8 20 3156K 800K nanslp 0:04 0.00% check_reload_status
368 root 1 -58 0 5716K 2844K bpf 0:04 0.00% tcpdump
5 root 1 -8 - 0K 8K - 0:03 0.00% g_down
47409 root 1 44 0 3524K 1848K RUN 0:03 0.00% top
4305 root 1 8 20 3492K 1492K wait 0:03 0.00% sh
369 root 1 -8 0 3156K 784K piperd 0:03 0.00% logger
2629 nobody 1 44 20 3156K 1304K select 0:03 0.00% dnsmasq
658 root 1 4 0 43784K 18392K accept 0:03 0.00% php
44 root 1 20 - 0K 8K syncer 0:02 0.00% syncer
12 root 1 -44 - 0K 8K WAIT 0:02 0.00% swi1: net
3 root 1 -8 - 0K 8K - 0:02 0.00% g_event
33 root 1 -8 - 0K 8K - 0:02 0.00% fdc0
22 root 1 8 - 0K 8K - 0:02 0.00% thread taskq
29 root 1 -64 - 0K 8K WAIT 0:01 0.00% irq14: ata0
639 root 1 4 0 45832K 19088K accept 0:01 0.00% php
618 root 1 4 0 45832K 19088K accept 0:01 0.00% php
37 root 1 8 - 0K 8K pftm 0:01 0.00% pfpurge
804 root 1 44 0 7012K 5912K select 0:01 0.00% bsnmpd
1255 root 1 44 0 7780K 3308K select 0:01 0.00% sshd
51 root 1 -8 - 0K 8K mdwait 0:01 0.00% md0
1141 root 1 44 0 3188K 1116K select 0:00 0.00% miniupnpd
45 root 1 -16 - 0K 8K sdflus 0:00 0.00% softdepflush
43 root 1 -4 - 0K 8K vlruwt 0:00 0.00% vnlru
42 root 1 -16 - 0K 8K psleep 0:00 0.00% bufdaemon
1156 root 1 8 0 3240K 1272K nanslp 0:00 0.00% cron
0 root 1 -16 0 0K 0K sched 0:00 0.00% swapper
2453 _dhcp 1 44 20 3156K 1316K select 0:00 0.00% dhclient
21 root 1 -28 - 0K 8K WAIT 0:00 0.00% swi5: +
41 root 1 171 ki31 0K 8K pollid 0:00 0.00% idlepoll
1183 root 1 8 0 3156K 900K nanslp 0:00 0.00% minicron
2656 dhcpd 1 44 20 3156K 2028K select 0:00 0.00% dhcpd -
1. you run iperf from firewall itself which is not good for test
2. I can't see that you have multiple processors - only cpu0, are you sure about SMP? -
Yes, I only have one CPU on this box. I tried the UP kernel, but performance was even much worse. I think ULE kernel is also good for UP.
I want to use pfsense as router. If I can't get wire speed for any nic, I'm not going to get wire speed routing performance. I tried run iperf between 2 hosts going through pfsense box, same 500mb/s.
Well, actually I route traffic through my HP Procurve switch, pfsense is my backup router and firewall to Internet. It'd be nice if it can route at wire speed.
-
Well… your system runs out of resources:
CPU: 0.0% user, 0.0% nice, 99.3% system, 0.0% interrupt, 0.7% idleTry disabling pf at all in System->Advanced, just out of curiousity.
-
Thanks a lot. After disabling PF, I'm getting wire speed again.
I thought PF puts very light load on system but I might be wrong. I have another 3G P4, I should be able to swap it and put more RAM in the system.
Again, thanks a lot.
-
RAM will not help you here. RAM is important for some packages + when you have tooo many active connections simultaneously.
You have decent (Intel) NICs.
You need more CPU power. It's just my opinion… -
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
26 root 1 -68 - 0K 8K - 0:39 66.06% em1 taskqtry disabling "Device Polling" in the System/Advanced tab if you have that option ticked, a quick google search yields similar problems with taskq and em(4) in some circumstances.
-
Polling was disabled. Thanks for the hints, I'll do more research on the issue.