@steveits The switch is a switch - so like with alle other switches:
1: If the switch is not in VLAN 801.2q mode, The ports acts as accessports by default and the traffic is just switched - no need to pass it into the CPU/uplink unless it’s destined for other devices (off current L2).
2: If you configure them as “discrete ports”, and put the switch in 802.1q mode, the ports are each configured as accessports in a given VLAN (usually unique) that is tunneled to the SOC on the uplink. (Note, you can have more than one port in the same VLAN tunnel), and all frames will have to pass through the uplink to be evaluated and routed across VLAN’s.
There is a performance difference between 4 switched ports (backed by a 2.5Gbps uplink port to the SOC), and four real NIC discrete ports. Not only is 2.5Gbps the theoretical throughput of routed traffic to/from the 4 switched ports - as opposed to 4Gbit for 4 individual interfaces but:
More importantly, small pfSense appliances does not have powerfull enough CPU cores to have a single core evalutate pfFilter rules (simple firewall rules) at Gbit wirespeed.
Since a interface queue is not properly multithreaded, the single CPU core performance becomes the bottleneck for throughput on each interface (aka - the switch uplink in this case).
FX: The SG-1100 that only has switched ports (All ports are seen on one uplink) has a max pffilter throughput of about 460Mbit.
The SG-2100 which has the same CPU as the SG-1100, but has a discrete NIC, and 4 switched ports through a Uplink, will do about 680Mbit in pfFilter if the traffic is passed from a switch port to the real NIC. This is because it has 2 queues - each can use its own CPU core, whereas the SG-1100 has only one Queue (the uplink), and is therefore mostly limited to one CPU core.
