But basically the LAGG algorithm is sending/receiving the file transfer on the same port on pfSense, so it's doing full duplex transfer.
I am not really sure but all depends on the configuration you made! You can also configure that one
LAN port is "doing" RX and the other is "doing" the TX part! And then you will be getting out;
1 GBit/s > TX
1 GBit/s > RX
And this might be then even 1 GBit/s and not 2 GBit/s! But for sure the entire LAG (LACP) is building
a aggregated 2 GBit/s fat pipe!
Now theoretically, the gigabit ethernet can handle 2000mbps total.
That is the exactly point where you are failing or made a so called thinking false in my eyes!
1 GBit/s line (cable) is able to send and receive 1 GBit/s over 4 adders of the cable in each direction
and this is then 1 GBit/s in each direction and not 2 GBit/s in one direction.
But I ran iperf between 2 machines using the simultaneous option, and the max I was about to get was about 450mbps both ways the same time. So not sure why?
If the technical and theoretical max throughput of a 1 GBit/s line is 125 MBit/s and with your LAG (LACP)
you will get out then in normal and as a max. 500 MBit/s (4 x 125 MBit/s) but you got 450 MBit/s + the
TCP/IP overhead that must be count on this on top you will be getting also nearly the macimum, or am I
wrong with this?
Anyhow when I transfer a file the other direction, the algorithm uses 2 ports on pfSense, so then I'm getting closer to 1Gb in that direction.
Then perhaps the network load you were producing with iPerf was not high enough perhaps I mean?
Either way, I think I will upgrade to 10Gbe with the Chelsio card, that should solve any Gb bottlenecks.
It is the best option as today in my eyes!!! The Chelsio card is fully offloading tasks such as VLANs based
on using an ASIC/FPGA on its NIC and it is better driver supported in pfSense! So you will be able to
fully unload from your pfSense box many TCP/IP based tasks and on top you will saving ports and
getting more throughput then now.