Ok... Here is 2 tests.. 1 where the networks are on their own physical interfaces
layout..
iperf server 192.168.9.10
iperf client 192.168.200.10
twophysicalnics.jpg
$ iperf3.exe -c 192.168.9.10 -B 192.168.200.10
warning: Ignoring nonsense TCP MSS 466688
Connecting to host 192.168.9.10, port 5201
[ 5] local 192.168.200.10 port 50165 connected to 192.168.9.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 108 MBytes 903 Mbits/sec
[ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 5] 2.00-3.00 sec 114 MBytes 954 Mbits/sec
[ 5] 3.00-4.00 sec 113 MBytes 949 Mbits/sec
[ 5] 4.00-5.00 sec 114 MBytes 957 Mbits/sec
[ 5] 5.00-6.00 sec 113 MBytes 950 Mbits/sec
[ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec
[ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 5] 8.00-9.00 sec 113 MBytes 948 Mbits/sec
[ 5] 9.00-10.00 sec 113 MBytes 950 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.10 GBytes 946 Mbits/sec sender
[ 5] 0.00-10.01 sec 1.10 GBytes 944 Mbits/sec receiver
iperf Done.
So that is maxing out gig.. Couldn't ask for anything more on gig wire..
Now here pfsense is routing between the networks over the same wire.. Same client and server machines - Just changed the switch config to put the client interface on different vlan. And put this vlan on the same physical interface used for vlan 9 (lan on pfsense) igb0
vlans-samephysical.jpg
$ iperf3.exe -c 192.168.9.10 -B 192.168.66.10
warning: Ignoring nonsense TCP MSS 466688
Connecting to host 192.168.9.10, port 5201
[ 5] local 192.168.66.10 port 50367 connected to 192.168.9.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 107 MBytes 895 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 5] 5.00-6.00 sec 111 MBytes 930 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec
[ 5] 7.00-8.00 sec 110 MBytes 925 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 934 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.08 GBytes 931 Mbits/sec sender
[ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec receiver
So not much difference because its duplex and no other traffic on the wire.. Bit of traffic maybe, the overhead of the vlan tags mentioned, etc..
But now sending traffic to the internet through pfsense through that same igb0 interface via speed test from client on that same vlan 9 network.. 500Mbps..
Now look at my iperf test..
$ iperf3.exe -c 192.168.9.10 -B 192.168.66.10
warning: Ignoring nonsense TCP MSS 466688
Connecting to host 192.168.9.10, port 5201
[ 5] local 192.168.66.10 port 50444 connected to 192.168.9.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 38.6 MBytes 324 Mbits/sec
[ 5] 1.00-2.00 sec 37.1 MBytes 311 Mbits/sec
[ 5] 2.00-3.00 sec 26.2 MBytes 220 Mbits/sec
[ 5] 3.00-4.00 sec 49.0 MBytes 411 Mbits/sec
[ 5] 4.00-5.00 sec 51.0 MBytes 428 Mbits/sec
[ 5] 5.00-6.00 sec 52.0 MBytes 436 Mbits/sec
[ 5] 6.00-7.00 sec 51.8 MBytes 434 Mbits/sec
[ 5] 7.00-8.00 sec 52.4 MBytes 439 Mbits/sec
[ 5] 8.00-9.00 sec 51.1 MBytes 429 Mbits/sec
[ 5] 9.00-10.00 sec 51.1 MBytes 429 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 460 MBytes 386 Mbits/sec sender
[ 5] 0.00-10.01 sec 460 MBytes 386 Mbits/sec receiver
iperf Done.
So there will be a performance hit when you share bandwidth of physical connection with vlans - because your sharing the capabilities of the interface... But without understanding your traffic flows, and amount of traffic that will be routed intervlan or using that interface going somewhere else, it hard to say if you will notice it or not..
Here is what I would suggest.. If you have the physical ports available on your switch and your router.. Then leverage them for your different networks so that vlans do not share physical ports..
If you do not have enough ports... Then put the vlans that do not talk to each other or use lower amounts of bandwidth on the same physical interface.. Example I put my wireless vlans on the same physical interface of pfsense... Since they would never be able to use full gig anyway, and they don't talk to each other..