VLAN routing overhead

johnpoz

If your going to do routing downstream of pfsense.. Your going to want pfsense to be connected to this downstream router via a transit network, and your going to have to modify your firewall rules, your outbound natting and create a gateway and routes to the networks that are behind your downstream router.

As to overloading pfsense? How much traffic are you moving what are the connection speeds into pfsense… So pfsense is a VM.. And your wanting to send traffic through pfsense to get to different vms? Why not just put these vms on the same network if they move a lot of data between each other and you don't care about firewalling between them.

Best would be if you could draw your current network - and point out the heavy data flow devices/networks and we can work out best way to design it. Downstream routers complicate the network quite a bit and most users don't seem to even know what a transit network is, etc. I would stay away from it -- devices that move lots of data between them, why can you not just put them on the same network so their conversations don't include pfsense at all.

netnewb

@johnpoz:

Best would be if you could draw your current network

It's very simple actually:

Internet –- Netgate SG-2440 --- CISCO SG200-18 ---- LAN
|
|
---- VLAN 17 Guest WiFi
|
|
--- VLAN 18 ( dokuwiki, jira )

And I want to do:

I wanted to put them in different VLANs to secure the devices at a firewall level because I thought it's simpler than at VM level. But now it seems more complicated to do that. One of the things I want to do, is to create a VM that archives mails on the NAS. The rest, I don't know yet but I don't think it'd move a lot of data frequently.

johnpoz

sg200 doesn't do L3 - sg200 is a L2 only..

So that throws your whole idea out of doing any routing on the switch.

So how many interfaces does your vm host have? If you have a lot of intervlan traffic. Sending that vlan up a trunk is now going to be a hairpin and yes /2 your available bandwidth of the physical link.

Your 2440 has 4 interfaces, so you have lan and opt1 and 2 so you could leverage opt1 and opt2 as uplink into pfsense and place your different networks/vlans that need to do a lot of talking to each other on the different uplinks.

But your going to need to do the same thing on your VM.. But if your wanting to firewall between your VMs you could always just do that all virtual with just a pfsense vm doing the firewalling between them. Ie pfsense becomes your downstream router..

jahonix

@johnpoz:

…and yes /2 your available bandwidth of the physical link...

BTW, are you sure about this?
We usually have full-duplex links so a 1Gb/s link is actually 1 up and 1 down, isn't it? A colleague recently asked this and I was a bit … :-X

netnewb

@johnpoz:

Your 2440 has 4 interfaces, so you have lan and opt1 and 2 so you could leverage opt1 and opt2 as uplink into pfsense and place your different networks/vlans that need to do a lot of talking to each other on the different uplinks.

This sounds like the easiest way to do it. I have 2 interfaces in the host and 4 in the NAS.

johnpoz

"We usually have full-duplex links so a 1Gb/s link is actually 1 up and 1 down"

Sure it is… you can tell your self that all day long ;) And it is full duplex.. but do a speed test from Device A and B on a switch in same vlan via say iperf.. What do you get high 800, low 900 mbps - you sure an the hell not going to see full Gig.. But why is that? Your on a full duplex connection - why are you not see 2gig??

Now put A on vlan 100, and B on vlan 200, and route them through your hairpin trunk and do the same test.. You still get 800 or 900mbps using iperf? Or do you see like half of that ;)

So your going to put your nas in 4 different networks?

johnpoz

Here maybe this makes it clearer.. And I answered your PM as well about it.. But this is for anyone else that might have the same question.. How may times is the packet on the wire.. How many packets can be on the wire at any one time?

So machine A wanting to talk to machine B… So syn gets sent up the wire.. So now it gets routed and that syn goes where? Back out the same wire with just a different tag on it.. So now the syn was on the wire how many times? 2 times vs before only being on it 1 time.. So what happens to the bandwidth when you /2 it since the packet was on the wire twice now vs once..

Now come back the ack.. So how many times that ack on the same wire.. When you use a trunk port, and traffic has to hairpin on that interface you double up the amount of traffic that is on that wire.. So when you double up the traffic what happens to the total bandwith you can see -- it gets /2...

P3R

I'm sorry but I still don't understand what you mean ???

@johnpoz:

So machine A wanting to talk to machine B… So syn gets sent up the wire.. So now it gets routed and that syn goes where? Back out the same wire with just a different tag on it..

It isn't the same wire as it goes back out on a different wire (or rather "channel" as it physically is more complicated within the TP-cable) on a full duplex connection.

Latency will increase due to the need for routing but assuming the router isn't a bandwidth bottleneck, I don't understand why bandwidth should be /2 as there are separate channels up and down (both capable of 1 gigabit concurrently) the hairpin cable with full duplex.

Another way to look at it:
If there are two separate interfaces in the router, each packet would travel the up channel in one physical cable and the down channel in the other cable.

With VLANs the packet will travel the up and down channels within the same physical cable but since up and down channels are independent of each other with full duplex, using the same physical cable shouldn't affect bandwidth.

johnpoz

How many up downs do you have in 1 physical cable your hung up on this duplex which doesn't mean anything - since your doubling up the amount of times a packet is on the same physical connection..

Lets look at it this way…

Here is your full duplex Your saying a packet can be on each one of those at the same time - great.. That is fantastic.. But how many times is the syn sent across?? Syn has to be sent over twice vs just once.. How many packets can be on a wire at any 1 time?? Only 1!!! So every packet has to be on the road twice.. What does that do to your bandwidth!!! /2

So as we are moving packets across your trunk how many packets in the full duplex connection can be on the wire?? 2 one on each path.. How many packets can be on the wire at the same time when each vlan has it own path.. 4!!

So lets say to move a file I have to put on the wire 100 packets.. Which is can do it faster? How many times do I need to send the packet over the physical wire - forget how many roads are in in phy wire.. Doesn't matter if 2 roads or 200 roads.. Only 1 packet can be on each road at anyone time..

You need to forget about the duplex because it does not really matter.. Does not matter how you look at the speed of the wire - if you want to think its 2gig in full duplex fine.. So when I am moving a file from machine A to machine B - why do I not get 2 gig ;) But sure ok you think the road can carry 2 gig.. Fine I am still putting the packets on that 2 gig road twice vs once.. So its /2 of the total.. If I had 3 vlans on it the trunk.. Only 1 packet can be on the road - so /3 if 5 vlans on the road - still only 1 packet can be on the wire at a time the the total bandwidth since shared if all talking at the same time would be /5..

trunkvsuplink.png_thumb

P3R

Okay, I think I finally understand what you mean.

@johnpoz:

Your saying a packet can be on each one of those at the same time…

I never said that. :(

A packet can be going in one direction while another packet can travel in the opposite direction at the same time. A single packet will be going up at 1 gigabit, be routed and then go down at 1 Gbps. A single packet will never ever travel the wire at 0.5 Gbps or less no matter how many VLANs there are in a trunk.

So every packet has to be on the road twice.. What does that do to your bandwidth!!! /2

In my opinion this explanation that you've tried many times now in different variations here is technically incorrect and it is what confused me. It isn't that a single packet will travel up to and down from the router in the same physical TP-cable that lead to less available bandwidth per VLAN in a trunk.

It's the fact that there are other VLANs competing for the bandwidth on the same shared medium that at times may lead to less than gigabit performance. On the other hand, that's a thing that should be considered when designing the network and if the VLANs are busy it should be dealt with by using link aggregation or a different design.

But sure ok you think the road can carry 2 gig..

I never said that either so I don't understand what gave you that idea. ???

Only people working in marketing departments claim a full duplex connection has bandwidth*2 and I'm very offended that you would think I'm in marketing. It's okay if you say that my kids are ugly but marketing, that's an insult! ;)

If I had 3 vlans on it the trunk.. Only 1 packet can be on the road - so /3 if 5 vlans on the road - still only 1 packet can be on the wire at a time the the total bandwidth since shared if all talking at the same time would be /5..

This is a very simple and logical explanation of what you mean. As the trunk is a shared medium, of course that will be the effect when all VLANS are fully loaded.

What you talk about is the absolute minimum bandwidth available per VLAN. When there's less than a 100 % load on any of the VLANs, the bandwidth in the other VLAN(s) will be higher, up to the maximum of 1 gigabit. In a connection with 2 VLANs, the per VLAN bandwidth will be somewhere between 0.5 Gbps and 1 Gbps. It will not always be 0.5 Gbps in each VLAN, which is the impression I got when you said that bandwidth is /2.

So in the old networking days, you never talked about 10 or 100 Mbps hubs? Instead you called all 8-port Fast Ethernet hubs 12.5 Mbps hubs and a 24 port was a 4.167 Mbps hub?

Maybe it's my limited understanding of this language that caused the confusion but I think that in this international forum I probably wasn't the only one to not understand your explanations.

jahonix

OK, I searched a bit and found this diagram which seems to clarify the situation:

I did NOT check this for being correct, but if both ends on all pairs of wires are transceivers, then it is perfectly clear that john is correct. Didn't know of the transceivers, always thought it would be two twisted-pairs each direction, thus my confusion.
Found this here: http://sqlblog.com/blogs/joe_chang/archive/2010/03/23/gigabit-and-full-duplex.aspx

P3R

@jahonix:

…but if both ends on all pairs of wires are transceivers, then it is perfectly clear that john is correct.

To be honest I can't tell from the posts in this thread if John agrees on that gigabit ethernet is 1 Gbps up and 1 Gbps down concurrently. Apparently you think that he doesn't agree to that and I certainly hope that he does.

Leaving VLANs aside for a moment, maybe John can clarify his view on that for us?

johnpoz

I agree that in duplex you can have up and down on the wire at the same time - ie a packet in both directions.. And if marketing wants to call it 2gpbs - that is marketing ;) Just like they market PHY for wireless…

My point is that there can only ever be 1 packet on a wire at any given time - be it the up wire (tx) or the down (rx) wire. If you trunk and have more than 1 vlan on the trunk, and these vlans talk to each other then the packet has to travel this wire twice once going to the router, once coming back to the switch. And to be honest doesn't even matter that only 1 packet can actually be on the wire at a time.. Its the fact that packet has to travel the same road twice. And the road is X wide, if you double up the times a packet travels this road then X /2 is just plain fact doesn't matter if you want to call it 2gbs pr not. When your machine moves a file it sure and the F does not get 2gbps does it.. When your wifi client is on wifi does it really get 300mbps ;) On 2x2 N connection??

Therefore does not matter how much bandwidth you say the wire has be it gig, 2 gig, 10 gig the fact that your hairpinning and packet has to travel the same road twice be it the same actual.. You guys are all thinking copper here -- fiber is full duplex as well ;) the tx and the rx.. Still same thing happens! when you hairpin to total available bandwidth is going to be /2 if device in vlan A is talking to vlan B and those are trunked on the same uplink.

Seem to be hung up on the actual phy make up of the road.. Which doesn't matter at all.. Its the fact that is a hairpin and your data is traveling the same road more than once that /2 the data that road can carry..

If your designing a network and you have vlan A that sends lots of data or gets lots of data to B.. I sure would not put them on the same uplink to your router.. A and B should be on their own uplinks or they should be in different trunks that uplink.. So lets say you have A, B, C and D for vlans and A and B do lots of chatter between them and moving data.. You don't put them on the same trunk you would to use 2 trunks and do it AC on 1 and BD on the other so that your not hairpinning when A and B talk. C or D could be using up the bandwidth when A and B are talking and still cause you a shared bandwidth problem. But atleast when A and B talk to each other its not a hairpin..

P3R

@johnpoz:

I agree that in duplex you can have up and down on the wire at the same time - ie a packet in both directions..

Great.

As we seem to have a hard time understanding each other I have to ask if you actively avoid the bandwidth part of my statement or if you agree on that as well. Does a full duplex gigabit ethernet allow traffic to flow at 1 Gbps up and 1 Gbps down at the same time?

In other words, can we agree on that a full duplex ethernet connection can be logically thought of as two independent one-way wires/channels/roads, with one leading up to a node (in this case a router) and the other leading down from it and since they are one-way paths packets can flow with 1 Gbps in both directions concurrently?

The above is very important here as that as far as I can tell was the part Jahonix got confused about.

kpa

1Gb/s link is 1Gb/s in both directions at the same time, the definition of full-duplex is that it's possible to transmit in both directions at the same time on the same medium. However, the two directions can never be mixed/merged together to form a faster link so both individual channels must be 1Gb/s or you would never be able to reach the transfer rates you're accustomed to on a gigabit network, only half of the rates if the opposite was true.

johnpoz

I think you getting tied up in the technical aspects of the link, yes its a tx and rx path that in theory can do 1gpbs in each direction. But that is not what you see in real world in how the exchange of information is done in tcp..

If I am pulling a file from a server B to server A - what is the speed at which I can move that file. Its going to be something less than 1Gbps.. If it was actually 2Gbps because its full duplex why do they not call it a 2Gbps connection ;)

Why does your interface never show its moving more than 1Gpbs? Even when you send a file and copy a file at the same time to the same server on the same switch why do you not see the interface go above its 1Gbps ;)

If your running gig - its a given that is full duplex. Yet a file from server A to server B is never going to be faster than your 1Gbps connection. Since I can only move a file at 1Gbps and now since I am sharing the trunk for both the traffic going to the router and then back down from the router for another vlan over the trunk the real world bandwidth that I can move my file is going to be approx /2 of the bandwidth I could see in moving the file from B to A if B and A were on the same switch.

They call it a 1Gbps interface for a reason ;) Since its "full duplex" if gig then given its full duplex so why do they not call it 2Gbps interface ;) If your using this interface as a trunk for the 2 vlans and both of these vlans are using the 1Gbps interface when moving data between server in vlan A to server or client in vlan B to be routed your now did a hairpin on the router interface and with a hairpin its going to be yes approx /2 the real world data flow if you had 2 different uplinks for these vlans or if each server was just moving data across a switch connected to their own ports in their own vlans - or even if the switch was routing the traffic since your not actually hairpinning a 1Gbps interface..

P3R

@johnpoz:

If it was actually 2Gbps because its full duplex why do they not call it a 2Gbps connection

I never claimed that a gigabit full duplex connection should be considered 2 Gbps but yet you've used that in every reply to me. Also instead of trying different ways to explain things when you're not getting through, you make long posts repeating the same things over and over.

I understand now that it's because you want to avoid any discussion and analysis of your /2 statement and instead aim to wear anybody down that question it. Since you're not interested in having a serious discussion I must now give up so mission accomplished for you. It's a pity though, since my reason for being here is to learn. :'(

jahonix

Oh boy, what have I started…

I did some reading of the IEEE 802.3ab standard and what's written is:
we have 1Gbit/s in each direction simultaneously.
(even though each of the 4 twisted pairs are bi-directional [the graphic above is correct], the transceivers can actually send and receive data at the same time. This is done with echo cancellation and adaptive equalization.)

As a thought experiment: take two 1Gbit switches and connect them with one CAT5e/6/7 cable.
Each side can send data at 1Gbit/s to the other side simultaneously. It doesn't matter if the packets carry VLAN ID bits or not. Think of UDP traffic so we don't have to account for ACK packets etc.
There is no /2 in the equation until now.

I'll digg further when I have the time to (which probably won't be until this weekend).

johnpoz

"Think of UDP traffic so we don't have to account for ACK packets etc."

Because yeah that what the enterprise and all users use to move files from their workstations to their NAS, etc.

Where did we say there was /2 on a switch.. So did you do that same thought experiment now between your vlans while you upload both those vlans up the trunk.. So up down the same wire - how much bandwidth do you get now?? Its a hairpin so its /2 the total be you want to discuss that its really 2 gig because its full duplex..

I am more than happy to have a discussion - what I am having a hard time understanding is what you do not understand about actual HAIRPIN?? It's a gig interface - you can only move data across it from one server to another server at a gig.. No I am not talking UDP across a switch port…. I am talking real world applications hairpin through a trunk to a router.. In this thought experiment do your udp test with that.. Do you still see your 2 gig total or not? Or did we now just cut it in 2 because your going up and then down the same wire twice vs once.. So you see what 1 gig do you not. So if in real world my clients only move 1 gig when doing a file copy, and then I hairpin it what happens to my gig - yeah 500mbps.. Be its full duplex or not..

If you can not grasp what a hairpin is - then I suggest you actually TEST it.. And see how you still get your gig on a hairpin because you think your really have 2 gig to play with because its full duplex..

So in our thought experiment and full duplex streaming UDP traffic at each other from 2 pc's at 1 gig each.. Ok there you have 2 gig.. This is the picture on the left.. red is 1 gbps, green is 1gbps

Now look at the picture at the right.. My PC still streaming 1 gig udp out and the other PC is streaming 1 gbps to the other machine.. But to get to each other they have to go through a hairpin via the trunk port to the router.. This connection still only has full duplex.. But how many red arrows are there, how many green arrows are there.. There is twice as many on the wire so there is you /2!!

So each side of the duplex has both gbs stream at the same time so now that uplink is 4gbs??? No since we know that is not the case -- guess the speed is going to be cut in half or /2 now isn't it..

So in a real world conversation while yes there is data flow in both directions.. Just like on the switch when I move a file via tcp you have packets with data, then you have your packets back with acks.. Flowing in both directions.. But my speed in my file transfer is something less then gig.. Now when I have to go up and then down the same wire to move those packets - what happens to the data.. Yeah that right /2..

Why this has had to drag on for so long I don't get.. It's a 1 gig connection, your moving the data up and then down the only 1 connection.. So how is it not just obvious that its /2??? You comprehend that if vlan A was trying to download from say the internet on that router, and vlan B was trying to download from the internet. That they would share the 1 gig uplink and their max speed would be /2 of whatever the internet speed was. But since they are talking each other which means up and down the same trunk that is also /2 but this doesn't make sense to you? That we have to break out the crayons is just freaking crazy!

internetupsametrunk.png_thumb

hairpinupatrunkduplex.png_thumb

P3R

@jahonix:

…the transceivers can actually send and receive data at the same time.

Yes, I found another graphic on a Cisco site that shows that it is 4*250 full duplex (send and receive at the same time) better but now we don't need that. :D

In my opinion going down to actual physical wire pairs may be more confusing (as it was to you). We only need to think of the connection as two channels, one up and one down and they're at 1 Gbps concurrently. If we settle on that, all this theory applies regardless of copper or optics are being used on the physical layer.

As a thought experiment: take two 1Gbit switches and connect them with one CAT5e/6/7 cable.
Each side can send data at 1Gbit/s to the other side simultaneously.

When those computers are in different VLANs and you hairpin that same traffic up to a router, the traffic sent from both computers need to share the channel up to the router and the traffic back out to both computers need to share the down link so throughput will be /2 on each stream. It will be two VLANs competing for access to the same up channel and also on the same down channel.

Let's do two more experiments:
#1. A more interesting case (if we really want to understand what happens) is when that same UDP traffic flows in only one direction, from one sender to a receiver. Then suddenly that stream will flow at 1 Gbps and definitely isn't /2.

#2. If in that same scenario we switch to TCP (still with a one-directional traffic flow), the throughput will effectively become the same as half duplex (minus the collisions since we don't need CSMA/CD in this full duplex connection) as the acks going back from the receiver need to share both the up link and then the down link with the actual data stream. A wild guess is that throughput will be around 900 Mbps. At least it should be far better than /2.

The /2 (or /x depending on how many VLANs we hairpin) will be true in a busy large network we're there's always an excess of traffic in all VLANs. On the other hand those networks are hopefully designed by knowledgeable people so less likely to be hairpinned or then at least using link aggregation or faster than gigabit interfaces to mitigate the effect of that shared connection.

If instead we talk about a home network, where probably the majority of the hairpins are deployed because it's there we often need to save money on router interfaces, the traffic is much more likely to be unevenly distributed between the VLANs. A DMZ may only see traffic occasionally when you want to check the ip cameras when nobody is at home (and therefore the other VLANs will have very little traffic). The administrative network will probably have a pretty low load. If we have a file server in a separate network, we're likely to see far better throughput than /2 when a single client up or download a file to it.