Is 2 NIC on one LAN on 2 different switch doable?
-
@frallard said in Is 2 NIC on one LAN on 2 different switch doable?:
minimize the traffic on the link between pfSense and the pSwitch. At the moment when my VMs are using the Internet, they need to go through the pSwitch on both link (pfSense <-> pSwitch <-> vSwitch on ESXi <-> Host). This affect all my internal services that also go through this link between ESXi and the pSwitch. To avoid that I would need to connect my pfSense on both LAN situations, one time on the pSwitch and one time on the vSwitch. I don't want to have a second set of IP addresses for my virtual LAN and route it in the pfSense for traffic between the two LAN. I was looking at LAGG interfaces, and I was wondering if that could work?
Ideally pfSense would only have one LAN interface in the firewall. Then the switch would understand that it's better to talk to that host on the virtual connection and that other host on the physical connection.
I hope my question is understandable enough and that I didn't ramble too much.
Oh, I'm using pfSense 2.4.4Yes if you do your LAGG on stacked switches i.e. you were using Cisco switches and stacking cables.
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/hardware/installation/guide/b_c3850_hig/b_c3850_hig_chapter_010.html#concept_731340C54C5C4974B300779F6D2728B6
With a stacked pair of Cisco switches the 1st port on the first stack would be int 1/1 and the corresponding port on the other switch would be 2/1, you could create a LAGG ( LACP ) of port 1/1 and 2/1.
Maybe draw a diagram of your setup.
-
@nogbadthebad said in Is 2 NIC on one LAN on 2 different switch doable?:
Maybe draw a diagram of your setup.
This!!! Why would traffic flowing between vms have anything to do with your physical network? So while yes your vms would go through your physical switch to get to your internet.. Not sure how your interVM traffic would effect that or be affected by your traffic to your physical world that it should matter.
You an always put your VMs on a different vswitch.. A drawing would help us all make sure we are on the same page in trying to figure out where you think you might have a bottleneck.
-
Thanks for your input, but one switch is physical and the other one is virtual, so no stacking available, this is why after reading a little more I figured out that LAGG was not a viable candidate. I'm looking into simple bridging and rely on STP. I'll draw a diagram in a minute.
Here is the diagram where you can see the actual working setup and what I want to add.
-
@frallard said in Is 2 NIC on one LAN on 2 different switch doable?:
I'm looking into simple bridging and rely on STP.
That is NOT going to be the best solution that is for damn sure!
Edit: This drawing isn't making a lot of sense.. Your vms are on what network this virtual lan of your is what for network 192.168.0/24 And that is tied to your pfsense (virtual router) that has a different lan tied to this physical network via different switch? Your doing this why... Why would you not just have both of these physical nics connected to your vswitch with pfsense having just a vnic into this vswitch?
-
@nogbadthebad said in Is 2 NIC on one LAN on 2 different switch doable?:
Why would traffic flowing between vms have anything to do with your physical network?It would not I know.
So while yes your vms would go through your physical switch to get to your internet.. Not sure how your interVM traffic would effect that or be affected by your traffic to your physical world that it should matter.
I have a VM that record OTA using 2 ATSC tuner that are on the physical side of things going into the virtual network. I have a VM that run my security camera that receive constant stream from the security cameras. This traffic on the physical link to my virtual lan prevent my VMs to get full Internet speed. I know it would not matter if my Internet were a 100 Mbps, but I get 1 Gbps and I would like mu VMs to get the best speed possible on Internet without impacting the physical link to my LAN.
You an always put your VMs on a different vswitch.. A drawing would help us all make sure we are on the same page in trying to figure out where you think you might have a bottleneck.
The drawing is created in the previous post.
-
Tie interfaces to the vswitch, connect pfsense lan to this vswitch and your vms on this vswitch = done.. Not understanindg why you have pfsense directly tied to the physical interface tied to the same lan??
-
Then my physical workstation would never be able to reach 1 Gbps because the physical link to virtual lan is busy with the traffic from the network cameras and ATSC tuners when in use and other traffic that flow from virtual to physical. I know because it was setup like you said. using iperf3 on the router I would get around 750 mbps while only all virtual and using Direct I/O I can do 950 mbps. I understand that the problem I have is only because my Internet connection is this fast, I would never have that problem if I had a reasonable Internet speed. I do know it is sooo overkill to have gigabit at home, but hey my ISP was proposing 50 Mbps or Gigabit for 10$ more per month. Nothing in between.
I have Multi-Link Trunking (Etherchannel) between my switch and ESXi, but that doesn't double the bandwidth, it distributes it. I was affected by the flowing traffic from physical to virtual when trying to reach maximum speed from my PC.
So I tough that there must be a way to present both leg to pfSense without going into a routing nightmare of having a subnet for virtual and an other for physical...
Bridging the two NIC would simply create kind of a switch inside pfSense for the LAN and then the 3 switches (pfSense bridge, vSwitch and pSwitch) connected together in triangle would hopefully figure out how to talk to each other and hopefully optimize where to send traffic...
I know, I know, I hear it : "hopefully" is really wishful thinking... But I don'T know about how STP works and how efficient it is at doing that...
Just trying to get the most out of my setup!
-
@frallard said in Is 2 NIC on one LAN on 2 different switch doable?:
and then rely on arp for the "decision" making of which nic to send packets to in order to reach the destination more efficiently?
ARP doesn't do that. All it does is determine the MAC address for an IP address.
I'm looking into simple bridging and rely on STP. I'll draw a diagram in a minute.
STP determines the best available layer 2 path to the root switch. Why do you think it would work here? Unless there is some change in the network configuration, that best path will not change. If I'm reading your diagram correctly, you want a connection between the VLAN and V3router. Unless that path is the lowest cost to the root switch, it will never be used. The STP root is determined by lowest priority and lowest MAC address. Where is that in your diagram?
-
@jknott
Thanks for your insight! I obviously don't know how to do it and this is why I'm asking for advise here. I was sharing my tough process and I can see now that it was flawed. From what I can understand, what I'm looking to do is not doable the way I want it to work. So I'll choose the way that work best for my need, so I'll stick to leaving the router connected to the physical switch using Direct I/O, because this is what gave me the best performance and have my VMs go back and forth on the switch to get to the WAN; they aren't the ones needing the most traffic anyway. -
This is what I ended up doing. Instead of trying the impossible, I created a virtual network DMZ. My router along with all my VMs, where I added a new nic, share this network. I changed the gateway configuration and my goal was reached. My local computers don't have to go through the router to get to the VMs and the VMs can benefit from having a virtual connection to the router all in virtual. My router has a forth dimension now! It lives in the virtual world and it also has physical assets that connects it to the real world.
-
@frallard said in Is 2 NIC on one LAN on 2 different switch doable?:
I do know it is sooo overkill to have gigabit at home
What? Gig is the MIN anyone should have at home... Anything else would be watching paint dry.. Shit I would have 10ge if wasn't so expensive ;)
I currently run 2 x1 ge interfaces on my PC and my Nas so I can use smb3 multichannel when moving files to and from my PC.. So vs the 113MBps I see 220MBps.
.
That's a copy of 8GB file -- so no gig is NOT overkill by any means at all. Being limited to 100mbps at home would be like being force to go back to dial up internet.
As to not reaching 1ge.. You would have both interfaces into your vswitch - so you would have 2 x 1ge connection that could be set for loadsharing. So no you wouldn't be limited to the 1ge with multiple sessions.
-
@johnpoz said in Is 2 NIC on one LAN on 2 different switch doable?:
Being limited to 100mbps at home would be like being force to go back to dial up internet.
Also, more and more gear now supports Gb, so might as well use it.
