Weird performance issue through LAN interface

[pwnell]

I have three interfaces on my pfsense box, LAN, WAN and WiFi.  All three NICs are 100baseTX FD.  LAN is connected to a small 5 port gigabit switch, WiFi is connected to an Apple Airport Extreme basestation and WAN is connected to my Cable modem.

I installed iperf and tested the throughput from a machine on my LAN to the LAN IP of the pfsense box.  I did the same in reverse.  Here are the results:

(pfsense is on 192.168.0.10)
–----------------------------------------------------------
Client connecting to 192.168.0.10, TCP port 5001
TCP window size: 65.0 KByte (default)

[  3] local 192.168.0.72 port 64105 connected with 192.168.0.10 port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0-10.0 sec    111 MBytes  92.8 Mbits/sec

This is awesome - almost 100Mbps.  I can live with that.  Problem is below:

Client connecting to 192.168.0.72, TCP port 5001
TCP window size: 65.0 KByte (default)
–----------------------------------------------------------
[  3] local 192.168.0.10 port 52830 connected with 192.168.0.72 port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0-10.2 sec  8.19 MBytes  6.73 Mbits/sec

So why would the performance drop so significantly when sending data from the pfsense box to the LAN machine?  All the machines in question has powerful CPU's.

[Guest]

What kind of NIC is it?  Do you have traffic shaping turned on?  Do you have device polling or checksum offloading turned on?  If you look at the device status do you see errors?  Dropped packets?

[pwnell]

@submicron:

What kind of NIC is it?  Do you have traffic shaping turned on?  Do you have device polling or checksum offloading turned on?  If you look at the device status do you see errors?  Dropped packets?

Ok you hit it on the nail - was traffic shaping.  What I do not get is why does it affect Firewall -> LAN transfers?  All my queues refer to WAN <-> LAN interactions.