STP and network
-
Could I do a kind of realistic test out of this before the actual going live?
Let's say I setup a pfSense in a closed environment, not connected to anything. I have one computer directly connected to the WAN-port with the computer having the IP 196.44.198.33 (/29-net) and no gateway-setting. This will kind of simulate my ISPs transit-network. I will then set the WAN interface on pfsense to be 196.44.198.34 (also /29-net), with that computer connected on WAN as GW.
On the LAN-side, I specify my current network, let's set it to be 4.4.4.2 (on /24) - I also create a VIP 4.4.4.1 that will serve as local gateway… I connect another computer, with IP 4.4.4.4 and specify 4.4.4.1 as default gateway. Now, I should only have to manage the outgoing NAT - Choose "WAN"-interface and choose the local VIP/GW under Address (and allow any on firewall) in order to ping 196.44.198.33. I understand that cluster requires a bit more, but baby steps are the way to go to understand this. Then I can test and basically do all the mistakes on my own ;) I'm very ready to test this, so please let me know as soon as possible if this could work!
-
Ok, I made it!
I didn't have to set up any NAT at all and I can ping from a computer on WAN side - and from LAN to WAN :)
Now, I have "faked" my ISP by letting a computer have an IP on the transport network. But I shouldn't actually need to change anything? Just use my ISPs IP as gw on pfsense WAN and I should be ready!
I don't see how DNS can be a problem either, I will continue to use my ISPs dns-servers and they are outside my network. As long as the Ip to their dns is allowed out, I shouldn't need to reconfigure any client computers after this change :)
Even LAG worked out of the box (I had to use active-passive since I didn't have any test LACP switch available).
-
But… The big question.. How do I do LACP from pfSense to both switches so that I get the setup I want. I have the two swiches that are stacked. But I want to have one cable from pfsenseLAG to SW1 and one cable to SW2. From what I understoon in this thread, I should be able to configure a LACP across both SW1 and SW2 now. So far, I have only found a way to do LACP on each of them at a time. I can of course switch fast between the two switches, but I'm missing a way to choose Port 47 on SW1 and Port48 on SW2 should be in same LACP.
"You would then put your 2 switches in a stack and setup a lacp lagg from pfsense to the switch stack with ports going to different switches in the stack."
So it is basically this I want to do.
-
So stack them and do that. Your switches have to be truly stackable (or support something like multi-chassis trunking), not some fake manage-all-as-one-switch marketing term stack.
Brocade ICX-6430:
lag Management dynamic id 81
ports ethernet 1/1/14 ethernet 2/1/14
primary-port 1/1/14
deploy
port-name NAS_LAGG0 ethernet 1/1/14
port-name NAS_LAGG1 ethernet 2/1/14
!Switch>sh lag id 81
Total number of LAGs: 2
Total number of deployed LAGs: 2
Total number of trunks created:2 (27 available)
LACP System Priority / ID: 1 / cc4e.24b3.68b8
LACP Long timeout: 90, default: 90
LACP Short timeout: 3, default: 3=== LAG "Management" ID 81 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/14 e 2/1/14
Port Count: 2
Primary Port: 1/1/14
Trunk Type: hash-based
LACP Key: 20081
Deployment: HW Trunk ID 1
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/14 Up Forward Full 1G 81 No 81 0 cc4e.24b3.68c5 NAS_LAGG0
2/1/14 Up Forward Full 1G 81 No 81 0 cc4e.24b3.68c5 NAS_LAGG1Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1/14 1 1 20081 Yes L Agg Syn Col Dis No No Ope
2/1/14 1 1 20081 Yes L Agg Syn Col Dis No No OpePartner Info and PDU Statistics
Port Partner Partner LACP LACP
System MAC Key Rx Count Tx Count
1/1/14 0cc4.7a47.7be2 203 2575780 2602883
2/1/14 0cc4.7a47.7be2 203 2575772 2602882Switch>sh stack
T=905d23h3m21.8: alone: standalone, D: dynamic cfg, S: static
ID Type Role Mac Address Pri State Comment
1 S ICX6430-24 active cc4e.24b3.68b8 128 local Ready
2 S ICX6430-24 standby cc4e.24b3.6978 0 remote Readyactive standby
+–-+ +---+
=2/3| 1 |2/1==2/3| 2 |2/1=
| +---+ +---+ |
| |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b3.68b8 -
I have stacked them - I think. They have a 10 Gbit fiber cable between them.
I can also access all the switches just with one IP thanks to the stacking. In the interface, I can choose between sw1, sw2 (that originally had each their IP). But I can't edit ports across both (choose one port from switch 1 and one from switch 2). From what I have found online, this is a feature called Cross-Stack and now I'm beginning to think I migth have purchased 4 switches that do stacking, but not this cross-stacking. This can also just be a cisco-word for it…
This is the unit:
http://us.dlink.com/us/en/business-solutions/switching/smart-switches/smartpro/dgs-1510-52x-52-port-gigabit-smartpro-switch.htmlI see that they have something called Physical stack (I have activated that). The switch shows a LED-number on each to show what number in the stack they are. In addition, I have also activated SIM I think that has to do with the shared IP. Maybe I have activated both a vitual method and the real thing at the same time ;) I hope it is just that.
Ok, I see I have to dive into documentation and see. Maybe I have overlook a setting or that it only done in cli.
-
What switches do you have?
-
DGS-1510-52X
-
At least there is a function called "Mirror" and I see I can choose ports on each of the stacked switches.. Not exactly what I'm looking for, but shows there are some integration..
-
I found the solution! Just had to create LACP with ID1 and only one member port. Then switch to next switch, create LACP and use ID1 on that as well :)
-
"Yes. And you need to adjust Outbound NAT so it NATs to the CARP VIP not to the interface addresses (for networks that might require NAT, that is)."
Since I didn't need to setup any NAT to get this working in non-carp mode apparently, I suspect I don't have to adjust anything in my scenario with carp either. Sounds like this would only complicate things. Remember that I use public static IPs on my LAN-side due to my type of webservers.
-
The inside addresses just need to be routed to the CARP VIP and not to one of the interface addresses.
-
Should I define a LACP-lag on the switch for each server or isn't that needed at all?
And as far I can tell, it works without LACP and I can remove one link and traffic still happens (but maybe switches get confused? Even though it says switch independent). But if it can creates weird situations, I wouldn't want to keep it that way.
From the Wiki for Centos, it seems like mode 4=802.3ad is the switch dependent mode on Linux: https://wiki.centos.org/TipsAndTricks/BondingInterfaces - but it will only have one active connection at a time.
But don't know what is best to choose in my case.
-
Should I define a LACP-lag on the switch for each server or isn't that needed at all?
I would. What happens when an entire switch fails. Understand that the LACP is Layer 2 redundancy, not layer 3. For layer 3 redundancy you don't need the LACP at all.
And as far I can tell, it works without LACP and I can remove one link and traffic still happens (but maybe switches get confused? Even though it says switch independent). But if it can creates weird situations, I wouldn't want to keep it that way.
When you remove what link?
From the Wiki for Centos, it seems like mode 4=802.3ad is the switch dependent mode on Linux: https://wiki.centos.org/TipsAndTricks/BondingInterfaces - but it will only have one active connection at a time.
Each side of an LACP link generally has a method of deciding what traffic it sends over what link. A combination of MAC address, IP address, and sometimes even port.
But don't know what is best to choose in my case.
Nor do I really.
-
I'm only using layer2 on this network.
"I would. What happens when an entire switch fails."
I don't mean that I should only connect to one switch, just if I should set up lag (either LACP or static) BOTH on switch and on server. There is a limit to number of LACP-team on the switch and a lot of administration to keep track of this. It is faster to just use the vendor-indepenendent solution, like mode=6 (balance-alb) on that wiki-page. I can connect each of the network-ports to different switch. "The interfaces are bonded in the Adaptive Load Balancing mode which supports both outgoing and recieving load balancing as well as failure support. This mode does not require any special switch support and is said to achieve load balancing by ARP negotiation."
"When you remove what link?"
One of the two network connections on the server.
-
Note that the switches now are in cluster, with 10G SPF+ between them.
I do see it might be an issue with having the non-switch configured method and not using the 802.3-method when there are one server against two different switches. This is how I have it today basically (but I don't have any major issues, at least none I can pinpoint to this).
I find a lot 802.3 post where it is first said it is switch-spesific, but once they see documentation, that says it is vendor-indepenendent, they are more open up for it. So it seems like a lot of opinions on this. Some think the active-backup method is better in that cases…
For me, it looks like the non-802.3 method kind of works like a failover HA. Maybe it could cause problems even in switch-cluster.
https://serverfault.com/questions/406672/link-bonding-across-multiple-switches
http://useopensource.blogspot.no/2010/02/linux-nic-teaming-recommendations.html
-
Well, your server issues are not pfSense issues. Need to do whatever it is that they support.
