Network performance issue

dilidolo

UDP or TCP?
What if you disable packet filtering?

TCP

PF rules for this internal network is set to allow everything now. still same.

Eugene

can you post top -S ?

dilidolo

last pid: 47571; load averages: 0.00, 0.00, 0.00 up 0+18:36:34 19:00:26
89 processes: 2 running, 74 sleeping, 13 waiting
CPU: 0.0% user, 0.4% nice, 0.8% system, 0.4% interrupt, 98.5% idle
Mem: 57M Active, 22M Inact, 32M Wired, 16K Cache, 19M Buf, 377M Free
Swap: 1024M Total, 1024M Free

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
11 root 1 171 ki31 0K 8K RUN 18.3H 100.00% idle: cpu0
28 root 1 -68 - 0K 8K WAIT 1:46 0.00% irq21: fxp0
25 root 1 -68 - 0K 8K - 1:03 0.00% em0 taskq
13 root 1 -32 - 0K 8K WAIT 1:00 0.00% swi4: clock sio
15 root 1 44 - 0K 8K - 0:18 0.00% yarrow
26 root 1 -68 - 0K 8K - 0:16 0.00% em1 taskq
614 root 1 4 0 5132K 3084K kqread 0:09 0.00% lighttpd
27 root 1 -68 - 0K 8K - 0:08 0.00% em2 taskq
361 root 1 8 0 3492K 1344K wait 0:07 0.00% sh
237 root 1 44 0 3268K 1152K select 0:06 0.00% syslogd
59242 root 1 4 0 40712K 14804K accept 0:06 0.00% php
31 root 1 0 - 0K 8K tzpoll 0:05 0.00% acpi_thermal
4 root 1 -8 - 0K 8K - 0:05 0.00% g_up
1181 root 1 8 20 3156K 800K nanslp 0:04 0.00% check_reload_status
368 root 1 -58 0 5716K 2836K bpf 0:04 0.00% tcpdump
5 root 1 -8 - 0K 8K - 0:03 0.00% g_down
369 root 1 -8 0 3156K 784K piperd 0:03 0.00% logger
4305 root 1 8 20 3492K 1492K wait 0:03 0.00% sh
658 root 1 4 0 43784K 18392K accept 0:03 0.00% php
2629 nobody 1 44 20 3156K 1304K select 0:02 0.00% dnsmasq
44 root 1 20 - 0K 8K syncer 0:02 0.00% syncer
12 root 1 -44 - 0K 8K WAIT 0:02 0.00% swi1: net
33 root 1 -8 - 0K 8K - 0:02 0.00% fdc0
3 root 1 -8 - 0K 8K - 0:02 0.00% g_event
22 root 1 8 - 0K 8K - 0:01 0.00% thread taskq
29 root 1 -64 - 0K 8K WAIT 0:01 0.00% irq14: ata0
639 root 1 4 0 45832K 19088K accept 0:01 0.00% php
618 root 1 4 0 45832K 19088K accept 0:01 0.00% php
37 root 1 8 - 0K 8K pftm 0:01 0.00% pfpurge
804 root 1 44 0 7012K 5880K select 0:01 0.00% bsnmpd
51 root 1 -8 - 0K 8K mdwait 0:01 0.00% md0
1255 root 1 44 0 7780K 3300K select 0:00 0.00% sshd
1141 root 1 44 0 3188K 1116K select 0:00 0.00% miniupnpd
45 root 1 -16 - 0K 8K sdflus 0:00 0.00% softdepflush
43 root 1 -4 - 0K 8K vlruwt 0:00 0.00% vnlru
42 root 1 -16 - 0K 8K psleep 0:00 0.00% bufdaemon
1156 root 1 8 0 3240K 1272K nanslp 0:00 0.00% cron
0 root 1 -16 0 0K 0K sched 0:00 0.00% swapper
2453 _dhcp 1 44 20 3156K 1316K select 0:00 0.00% dhclient
21 root 1 -28 - 0K 8K WAIT 0:00 0.00% swi5: +
41 root 1 171 ki31 0K 8K pollid 0:00 0.00% idlepoll
1183 root 1 8 0 3156K 900K nanslp 0:00 0.00% minicron
1286 root 1 20 0 4532K 2452K pause 0:00 0.00% tcsh
38 root 1 -16 - 0K 8K psleep 0:00 0.00% pagedaemon
2656 dhcpd 1 44 20 3156K 2028K select 0:00 0.00% dhcpd

Thanks for the response.

Eugene

No, it is impossible.
I mean top -S on the firewall through which you are doing traffic test.

dilidolo

last pid: 57439; load averages: 0.48, 0.16, 0.05 up 0+19:40:41 20:04:33
95 processes: 3 running, 79 sleeping, 13 waiting
CPU: 0.0% user, 0.0% nice, 99.3% system, 0.0% interrupt, 0.7% idle
Mem: 58M Active, 22M Inact, 33M Wired, 16K Cache, 19M Buf, 375M Free
Swap: 1024M Total, 1024M Free

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
26 root 1 -68 - 0K 8K - 0:39 66.06% em1 taskq
11 root 1 171 ki31 0K 8K RUN 19.4H 17.09% idle: cpu0
57287 root 4 99 0 4864K 2024K RUN 0:03 15.19% iperf
28 root 1 -68 - 0K 8K WAIT 1:47 0.00% irq21: fxp0
13 root 1 -32 - 0K 8K WAIT 1:04 0.00% swi4: clock sio
25 root 1 -68 - 0K 8K - 1:03 0.00% em0 taskq
15 root 1 44 - 0K 8K - 0:18 0.00% yarrow
614 root 1 4 0 5132K 3084K kqread 0:09 0.00% lighttpd
27 root 1 -68 - 0K 8K - 0:08 0.00% em2 taskq
361 root 1 8 0 3492K 1344K wait 0:07 0.00% sh
237 root 1 44 0 3268K 1152K select 0:07 0.00% syslogd
59242 root 1 4 0 40712K 14804K accept 0:06 0.00% php
31 root 1 0 - 0K 8K tzpoll 0:05 0.00% acpi_thermal
4 root 1 -8 - 0K 8K - 0:05 0.00% g_up
1181 root 1 8 20 3156K 800K nanslp 0:04 0.00% check_reload_status
368 root 1 -58 0 5716K 2844K bpf 0:04 0.00% tcpdump
5 root 1 -8 - 0K 8K - 0:03 0.00% g_down
47409 root 1 44 0 3524K 1848K RUN 0:03 0.00% top
4305 root 1 8 20 3492K 1492K wait 0:03 0.00% sh
369 root 1 -8 0 3156K 784K piperd 0:03 0.00% logger
2629 nobody 1 44 20 3156K 1304K select 0:03 0.00% dnsmasq
658 root 1 4 0 43784K 18392K accept 0:03 0.00% php
44 root 1 20 - 0K 8K syncer 0:02 0.00% syncer
12 root 1 -44 - 0K 8K WAIT 0:02 0.00% swi1: net
3 root 1 -8 - 0K 8K - 0:02 0.00% g_event
33 root 1 -8 - 0K 8K - 0:02 0.00% fdc0
22 root 1 8 - 0K 8K - 0:02 0.00% thread taskq
29 root 1 -64 - 0K 8K WAIT 0:01 0.00% irq14: ata0
639 root 1 4 0 45832K 19088K accept 0:01 0.00% php
618 root 1 4 0 45832K 19088K accept 0:01 0.00% php
37 root 1 8 - 0K 8K pftm 0:01 0.00% pfpurge
804 root 1 44 0 7012K 5912K select 0:01 0.00% bsnmpd
1255 root 1 44 0 7780K 3308K select 0:01 0.00% sshd
51 root 1 -8 - 0K 8K mdwait 0:01 0.00% md0
1141 root 1 44 0 3188K 1116K select 0:00 0.00% miniupnpd
45 root 1 -16 - 0K 8K sdflus 0:00 0.00% softdepflush
43 root 1 -4 - 0K 8K vlruwt 0:00 0.00% vnlru
42 root 1 -16 - 0K 8K psleep 0:00 0.00% bufdaemon
1156 root 1 8 0 3240K 1272K nanslp 0:00 0.00% cron
0 root 1 -16 0 0K 0K sched 0:00 0.00% swapper
2453 _dhcp 1 44 20 3156K 1316K select 0:00 0.00% dhclient
21 root 1 -28 - 0K 8K WAIT 0:00 0.00% swi5: +
41 root 1 171 ki31 0K 8K pollid 0:00 0.00% idlepoll
1183 root 1 8 0 3156K 900K nanslp 0:00 0.00% minicron
2656 dhcpd 1 44 20 3156K 2028K select 0:00 0.00% dhcpd

Eugene

1. you run iperf from firewall itself which is not good for test
2. I can't see that you have multiple processors - only cpu0, are you sure about SMP?

dilidolo

Yes, I only have one CPU on this box. I tried the UP kernel, but performance was even much worse. I think ULE kernel is also good for UP.

I want to use pfsense as router. If I can't get wire speed for any nic, I'm not going to get wire speed routing performance. I tried run iperf between 2 hosts going through pfsense box, same 500mb/s.

Well, actually I route traffic through my HP Procurve switch, pfsense is my backup router and firewall to Internet. It'd be nice if it can route at wire speed.

Eugene

Well… your system runs out of resources:
CPU: 0.0% user, 0.0% nice, 99.3% system, 0.0% interrupt, 0.7% idle

Try disabling pf at all in System->Advanced, just out of curiousity.

dilidolo

Thanks a lot. After disabling PF, I'm getting wire speed again.

I thought PF puts very light load on system but I might be wrong. I have another 3G P4, I should be able to swap it and put more RAM in the system.

Again, thanks a lot.

Eugene

RAM will not help you here. RAM is important for some packages + when you have tooo many active connections simultaneously.
You have decent (Intel) NICs.
You need more CPU power. It's just my opinion…

Slam

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
26 root 1 -68 - 0K 8K - 0:39 66.06% em1 taskq

try disabling "Device Polling" in the System/Advanced tab if you have that option ticked, a quick google search yields similar problems with taskq and em(4) in some circumstances.

http://www.google.co.uk/search?hl=en&client=firefox-a&channel=s&rls=org.mozilla%3Aen-GB%3Aofficial&q=taskq%2Bem%2Bperformance&btnG=Search&meta=&aq=&oq=taskq%2Bem%2Bperformanc

dilidolo

Polling was disabled. Thanks for the hints, I'll do more research on the issue.