VLAN routing overhead

marvosa

Would need more details on your setup, but if your VLANs are terminated on PFsense and there's only 1 link to your switch then it's possible that intervlan traffic could saturate that link

To answer your question, no, the traffic won't overload PFsense itself, but it can slow down your network by saturating the link between PFsense and your switch.

jahonix

Obviously, your switch is managed. Can it do L3 routing as well?
If your NAS must not reside on the same L2 layer where your hosts are then this would speed up transfers. Routing in a switch is nearly wirespeed.

netnewb

@jahonix:

Obviously, your switch is managed. Can it do L3 routing as well?
If your NAS must not reside on the same L2 layer where your hosts are then this would speed up transfers. Routing in a switch is nearly wirespeed.

The switch is a Cisco SG200-18 . I still have some trouble understanding layers, but I'm pretty sure it does L3 routing. If I undestand correctly, if I do the routing at switch level, it means I'd have to allow access from one VLAN to another VLAN, but on all ports. Right? But the advantage is that traffic wouldn't pass through pfsense and saturate the link.

jahonix

Correct.

netnewb

@jahonix:

Correct.

Thanks!

johnpoz

If your going to do routing downstream of pfsense.. Your going to want pfsense to be connected to this downstream router via a transit network, and your going to have to modify your firewall rules, your outbound natting and create a gateway and routes to the networks that are behind your downstream router.

As to overloading pfsense? How much traffic are you moving what are the connection speeds into pfsense… So pfsense is a VM.. And your wanting to send traffic through pfsense to get to different vms? Why not just put these vms on the same network if they move a lot of data between each other and you don't care about firewalling between them.

Best would be if you could draw your current network - and point out the heavy data flow devices/networks and we can work out best way to design it. Downstream routers complicate the network quite a bit and most users don't seem to even know what a transit network is, etc. I would stay away from it -- devices that move lots of data between them, why can you not just put them on the same network so their conversations don't include pfsense at all.

netnewb

@johnpoz:

Best would be if you could draw your current network

It's very simple actually:

Internet –- Netgate SG-2440 --- CISCO SG200-18 ---- LAN
|
|
---- VLAN 17 Guest WiFi
|
|
--- VLAN 18 ( dokuwiki, jira )

And I want to do:

I wanted to put them in different VLANs to secure the devices at a firewall level because I thought it's simpler than at VM level. But now it seems more complicated to do that. One of the things I want to do, is to create a VM that archives mails on the NAS. The rest, I don't know yet but I don't think it'd move a lot of data frequently.

johnpoz

sg200 doesn't do L3 - sg200 is a L2 only..

So that throws your whole idea out of doing any routing on the switch.

So how many interfaces does your vm host have? If you have a lot of intervlan traffic. Sending that vlan up a trunk is now going to be a hairpin and yes /2 your available bandwidth of the physical link.

Your 2440 has 4 interfaces, so you have lan and opt1 and 2 so you could leverage opt1 and opt2 as uplink into pfsense and place your different networks/vlans that need to do a lot of talking to each other on the different uplinks.

But your going to need to do the same thing on your VM.. But if your wanting to firewall between your VMs you could always just do that all virtual with just a pfsense vm doing the firewalling between them. Ie pfsense becomes your downstream router..

jahonix

@johnpoz:

…and yes /2 your available bandwidth of the physical link...

BTW, are you sure about this?
We usually have full-duplex links so a 1Gb/s link is actually 1 up and 1 down, isn't it? A colleague recently asked this and I was a bit … :-X

netnewb

@johnpoz:

Your 2440 has 4 interfaces, so you have lan and opt1 and 2 so you could leverage opt1 and opt2 as uplink into pfsense and place your different networks/vlans that need to do a lot of talking to each other on the different uplinks.

This sounds like the easiest way to do it. I have 2 interfaces in the host and 4 in the NAS.

johnpoz

"We usually have full-duplex links so a 1Gb/s link is actually 1 up and 1 down"

Sure it is… you can tell your self that all day long ;) And it is full duplex.. but do a speed test from Device A and B on a switch in same vlan via say iperf.. What do you get high 800, low 900 mbps - you sure an the hell not going to see full Gig.. But why is that? Your on a full duplex connection - why are you not see 2gig??

Now put A on vlan 100, and B on vlan 200, and route them through your hairpin trunk and do the same test.. You still get 800 or 900mbps using iperf? Or do you see like half of that ;)

So your going to put your nas in 4 different networks?

johnpoz

Here maybe this makes it clearer.. And I answered your PM as well about it.. But this is for anyone else that might have the same question.. How may times is the packet on the wire.. How many packets can be on the wire at any one time?

So machine A wanting to talk to machine B… So syn gets sent up the wire.. So now it gets routed and that syn goes where? Back out the same wire with just a different tag on it.. So now the syn was on the wire how many times? 2 times vs before only being on it 1 time.. So what happens to the bandwidth when you /2 it since the packet was on the wire twice now vs once..

Now come back the ack.. So how many times that ack on the same wire.. When you use a trunk port, and traffic has to hairpin on that interface you double up the amount of traffic that is on that wire.. So when you double up the traffic what happens to the total bandwith you can see -- it gets /2...

P3R

I'm sorry but I still don't understand what you mean ???

@johnpoz:

So machine A wanting to talk to machine B… So syn gets sent up the wire.. So now it gets routed and that syn goes where? Back out the same wire with just a different tag on it..

It isn't the same wire as it goes back out on a different wire (or rather "channel" as it physically is more complicated within the TP-cable) on a full duplex connection.

Latency will increase due to the need for routing but assuming the router isn't a bandwidth bottleneck, I don't understand why bandwidth should be /2 as there are separate channels up and down (both capable of 1 gigabit concurrently) the hairpin cable with full duplex.

Another way to look at it:
If there are two separate interfaces in the router, each packet would travel the up channel in one physical cable and the down channel in the other cable.

With VLANs the packet will travel the up and down channels within the same physical cable but since up and down channels are independent of each other with full duplex, using the same physical cable shouldn't affect bandwidth.

johnpoz

How many up downs do you have in 1 physical cable your hung up on this duplex which doesn't mean anything - since your doubling up the amount of times a packet is on the same physical connection..

Lets look at it this way…

Here is your full duplex Your saying a packet can be on each one of those at the same time - great.. That is fantastic.. But how many times is the syn sent across?? Syn has to be sent over twice vs just once.. How many packets can be on a wire at any 1 time?? Only 1!!! So every packet has to be on the road twice.. What does that do to your bandwidth!!! /2

So as we are moving packets across your trunk how many packets in the full duplex connection can be on the wire?? 2 one on each path.. How many packets can be on the wire at the same time when each vlan has it own path.. 4!!

So lets say to move a file I have to put on the wire 100 packets.. Which is can do it faster? How many times do I need to send the packet over the physical wire - forget how many roads are in in phy wire.. Doesn't matter if 2 roads or 200 roads.. Only 1 packet can be on each road at anyone time..

You need to forget about the duplex because it does not really matter.. Does not matter how you look at the speed of the wire - if you want to think its 2gig in full duplex fine.. So when I am moving a file from machine A to machine B - why do I not get 2 gig ;) But sure ok you think the road can carry 2 gig.. Fine I am still putting the packets on that 2 gig road twice vs once.. So its /2 of the total.. If I had 3 vlans on it the trunk.. Only 1 packet can be on the road - so /3 if 5 vlans on the road - still only 1 packet can be on the wire at a time the the total bandwidth since shared if all talking at the same time would be /5..

trunkvsuplink.png_thumb

P3R

Okay, I think I finally understand what you mean.

@johnpoz:

Your saying a packet can be on each one of those at the same time…

I never said that. :(

A packet can be going in one direction while another packet can travel in the opposite direction at the same time. A single packet will be going up at 1 gigabit, be routed and then go down at 1 Gbps. A single packet will never ever travel the wire at 0.5 Gbps or less no matter how many VLANs there are in a trunk.

So every packet has to be on the road twice.. What does that do to your bandwidth!!! /2

In my opinion this explanation that you've tried many times now in different variations here is technically incorrect and it is what confused me. It isn't that a single packet will travel up to and down from the router in the same physical TP-cable that lead to less available bandwidth per VLAN in a trunk.

It's the fact that there are other VLANs competing for the bandwidth on the same shared medium that at times may lead to less than gigabit performance. On the other hand, that's a thing that should be considered when designing the network and if the VLANs are busy it should be dealt with by using link aggregation or a different design.

But sure ok you think the road can carry 2 gig..

I never said that either so I don't understand what gave you that idea. ???

Only people working in marketing departments claim a full duplex connection has bandwidth*2 and I'm very offended that you would think I'm in marketing. It's okay if you say that my kids are ugly but marketing, that's an insult! ;)

If I had 3 vlans on it the trunk.. Only 1 packet can be on the road - so /3 if 5 vlans on the road - still only 1 packet can be on the wire at a time the the total bandwidth since shared if all talking at the same time would be /5..

This is a very simple and logical explanation of what you mean. As the trunk is a shared medium, of course that will be the effect when all VLANS are fully loaded.

What you talk about is the absolute minimum bandwidth available per VLAN. When there's less than a 100 % load on any of the VLANs, the bandwidth in the other VLAN(s) will be higher, up to the maximum of 1 gigabit. In a connection with 2 VLANs, the per VLAN bandwidth will be somewhere between 0.5 Gbps and 1 Gbps. It will not always be 0.5 Gbps in each VLAN, which is the impression I got when you said that bandwidth is /2.

So in the old networking days, you never talked about 10 or 100 Mbps hubs? Instead you called all 8-port Fast Ethernet hubs 12.5 Mbps hubs and a 24 port was a 4.167 Mbps hub?

Maybe it's my limited understanding of this language that caused the confusion but I think that in this international forum I probably wasn't the only one to not understand your explanations.

jahonix

OK, I searched a bit and found this diagram which seems to clarify the situation:

I did NOT check this for being correct, but if both ends on all pairs of wires are transceivers, then it is perfectly clear that john is correct. Didn't know of the transceivers, always thought it would be two twisted-pairs each direction, thus my confusion.
Found this here: http://sqlblog.com/blogs/joe_chang/archive/2010/03/23/gigabit-and-full-duplex.aspx

P3R

@jahonix:

…but if both ends on all pairs of wires are transceivers, then it is perfectly clear that john is correct.

To be honest I can't tell from the posts in this thread if John agrees on that gigabit ethernet is 1 Gbps up and 1 Gbps down concurrently. Apparently you think that he doesn't agree to that and I certainly hope that he does.

Leaving VLANs aside for a moment, maybe John can clarify his view on that for us?

johnpoz

I agree that in duplex you can have up and down on the wire at the same time - ie a packet in both directions.. And if marketing wants to call it 2gpbs - that is marketing ;) Just like they market PHY for wireless…

My point is that there can only ever be 1 packet on a wire at any given time - be it the up wire (tx) or the down (rx) wire. If you trunk and have more than 1 vlan on the trunk, and these vlans talk to each other then the packet has to travel this wire twice once going to the router, once coming back to the switch. And to be honest doesn't even matter that only 1 packet can actually be on the wire at a time.. Its the fact that packet has to travel the same road twice. And the road is X wide, if you double up the times a packet travels this road then X /2 is just plain fact doesn't matter if you want to call it 2gbs pr not. When your machine moves a file it sure and the F does not get 2gbps does it.. When your wifi client is on wifi does it really get 300mbps ;) On 2x2 N connection??

Therefore does not matter how much bandwidth you say the wire has be it gig, 2 gig, 10 gig the fact that your hairpinning and packet has to travel the same road twice be it the same actual.. You guys are all thinking copper here -- fiber is full duplex as well ;) the tx and the rx.. Still same thing happens! when you hairpin to total available bandwidth is going to be /2 if device in vlan A is talking to vlan B and those are trunked on the same uplink.

Seem to be hung up on the actual phy make up of the road.. Which doesn't matter at all.. Its the fact that is a hairpin and your data is traveling the same road more than once that /2 the data that road can carry..

If your designing a network and you have vlan A that sends lots of data or gets lots of data to B.. I sure would not put them on the same uplink to your router.. A and B should be on their own uplinks or they should be in different trunks that uplink.. So lets say you have A, B, C and D for vlans and A and B do lots of chatter between them and moving data.. You don't put them on the same trunk you would to use 2 trunks and do it AC on 1 and BD on the other so that your not hairpinning when A and B talk. C or D could be using up the bandwidth when A and B are talking and still cause you a shared bandwidth problem. But atleast when A and B talk to each other its not a hairpin..

P3R

@johnpoz:

I agree that in duplex you can have up and down on the wire at the same time - ie a packet in both directions..

Great.

As we seem to have a hard time understanding each other I have to ask if you actively avoid the bandwidth part of my statement or if you agree on that as well. Does a full duplex gigabit ethernet allow traffic to flow at 1 Gbps up and 1 Gbps down at the same time?

In other words, can we agree on that a full duplex ethernet connection can be logically thought of as two independent one-way wires/channels/roads, with one leading up to a node (in this case a router) and the other leading down from it and since they are one-way paths packets can flow with 1 Gbps in both directions concurrently?

The above is very important here as that as far as I can tell was the part Jahonix got confused about.

kpa

1Gb/s link is 1Gb/s in both directions at the same time, the definition of full-duplex is that it's possible to transmit in both directions at the same time on the same medium. However, the two directions can never be mixed/merged together to form a faster link so both individual channels must be 1Gb/s or you would never be able to reach the transfer rates you're accustomed to on a gigabit network, only half of the rates if the opposite was true.