Another topology sanity check...
-
Hi. I am having some issues with inter-vlan traffic. Crossing over VLANs I can't seem to top 910 Mbps on 1 Gbps links using a 10 Gbps SFP+ X710-DA2 (tried both single and LACP) where I trunk all the VLANs. The CPU is a Xeon E-2278G (so grossly overpowered). I experimented separating out the VLANs to physical interfaces since I have 8 i210 NICs on the motherboard, trying to eliminate the hairpin (though I didn't think it would be an issue with the 10G and sure enough, 941 Mbps on the 1 Gbps devices.
This is my current setup and what I am thinking of moving to
But I am not a network expert and not sure if this is the correct way to layout the switches, uplinks, etc. Any advice or thoughts? Thanks!
Also considering getting rid of the "Homelab" VLAN and getting rid of all the LACP connections, as its overkill 1000x in Homelab.. and making a more simple layout.. Basically all my trusted devices and homelab just on the native VLAN essentially.
But I am not sure if I will have a problem with multiple uplinks for the different subnets the way I have have it laid out.
-
From a performance standpoint, you could create a transit network between PFsense and your core switch and then move your VLANs to the core switch. This way, inter-vlan traffic is handled by the switch instead of having to traverse PFsense.
But TBH, as long as you're using VLANs, 910 Mbps (110 MB/s) is probably about as close to wire speed as you're gonna get.
Others can chime in with their experiences, but I can't see getting much more throughput than you already have without maybe experimenting with jumbo frames.
On a side note, a quick google search suggests tweaking some client-side settings like flow control, receive side scaling, interrupt moderation, transmit/receive buffers, TCP offloading, etc may offer some benefit as well.
-
@marvosa yea, unfortunately my switches are L2 only, so I don't think inter-vlan routing on the switch is going work for me. Interesting to note VLAN overhead. I didn't think it was that much, but frankly I don't have much experience with VLANs and this is my first attempt at VLANs on a network I control. Thanks for your feedback. Definitely helps me understand and have some base expectations with routing VLANs through pfSense :)