Switching from igc1 -> ic0 as physical interface starts well and then collapses
-
Hi guys
Strange...
I have a Topton with 4 x 2.5GbE Cat interface and 2 x 10GbE SFP cages.
Currently port igc1 has 172.16.10.1 assigned (2nd copper). This is then patched into my core switch Unifi USW Pro Max with 172.16.10.2 assigned to port 24.
172.16.10.0/24 being the main lan, the main internal network port on the pfSense.
over this physical trunk I run the following vlan's.
vlan20 172.16.20.0/24 - Wifi network with AP's
vlan30 172.16.30.0/24 - ToBe ProxMox Cluster interconnect
vlan40 172.16.40.0/24 - ProxMox IO
vlan100 172.16.100.0/24 - IoT Wifi network over AP'sI've now repeatedly tried to switch/move all traffic on pfSense from igc1 (on port 24)-> ix0 (on port 25)
Steps followed....
move vlan40 from igc1 -> ix0 and
move vlan30 from igc1 -> ix0 and
move vlan100 from igc1 -> ix0 and
move vlan20 from igc1 -> ix0Last step move/reconfigure lan from igc1 onto ix0 making ix0 the master/parent for all now.
At this port all is still good, stables. I can even go onto my Unifi and disable port 24. all good... Now the strange, the min i now unpatch port24 then all comes apart.
G
Black is 1GbE Cat5e
Blue is 2.5GbE Cat6
Red is to be 10GB, the pfSense to Unifi Switch link is to be a 10GbE DACNote, I have a Unifi Aggregation on the way, all the Proxmox Topton devices with 10GbE will patch into the aggregation (to be 172.16.10.3), the aggregation will then patch into port 26 on the ProMax.
Each ProxMox Topton will have a 2.5GbE onto Lan/172.16.10.0/24
Each ProxMox Topton will have vlan30 and vlan40 exposed via the 10GbE links.This was the attempt...
A view of the current working, you will see the pfsense to ProMax is 2.5GbE/Cat 6 based.
igc1
172.16.10.1 - Core Lan
172.16.20.1 - vLan20
172.16.30.1 - vLan30
172.16.40.1 - vLan40
172.16.100.1 - vlan100This is the to be/planned
I have considered running only vlan30/40 on 10GbE between prSense and port25 and leaving lan/vlan20/vlan100 on the 2.5GbE link between igc1 and port 24. - no luck.
Different view of the to be thinking
bolded text -
@georgelza said in Switching from igc1 -> ic0 as physical interface starts well and then collapses:
At this port all is still good, stables. I can even go onto my Unifi and disable port 24. all good... Now the strange, the min i now unpatch port24 then all comes apart
Do you mean that when you unplug the cable from port 24? What exactly happens then?
You mentioned that you have 172.16.10.2 assigned to port 24. Are you saying that you set that in the switch itself? I sort of think that the switch should still work, even if you are not connected to that specific port, but who knows. I'm not sure what happens from the Unifi Controller side when it looses the connetion?
If you set the IP in the swtich, I'd suggest to handle that from pfsense instead... Then it shouldn't matter which ports are connected or not... -
H there.
Yes 172.16.10.2 is static assigned to the switch, need to check... hight have something here... need to see if it is assigned to the port24... meaning when we flip the traffic over to the fiber port it's not 172.16.10.2 anymore...
it's just very weird, with everything (and that might limited as far as a network engineer level is concerned, but exactly idiot level) this should work...
Unstable mean, it starts getting collisions, eventually with the AP.s and other client switches loosing their identity and going into port isolated.
my DHCP is run on the pfSense, so something happens there, that then runs downhill.
my thinking something happens and comms with the ProMax is lost, resulting in client devices on the ProMax loosing their identity, also as most of my Wifi devices sit on 172.16.20.0/24 and the physical servers etc on 172.16.10.0/24 we have routing and inter lan/vlan that needs to happen... so with whatever happens up stream,, that routing ability is lost...
The unifi controller sits on my TrueNAS on 172.16.10.24 as a container... so when this internal coming part happens it is isolated/lost also it seems...
-
@Gblenn so figured out the up assigned on the device/switch to use on it's uplink port, whatever that is...
G
-
i'm thinking about moving Unifi Controller onto a small physical device and patching that directly into the pfSense, sitting on 172.16.10.0/net working...
Just need to see what the best option is for Unifi controller... as far as deployment is concerned, easiest would be a unix Os with Docker and then run as a container, allowing me potentially to run other containers on the physical device also that has a high level of importance.Making it more independent from the larger network...
G
-
@georgelza said in Switching from igc1 -> ic0 as physical interface starts well and then collapses:
i'm thinking about moving Unifi Controller onto a small physical device and patching that directly into the pfSense
I run it as a Proxmox VM. It work well for me. You may find it easy to run that way as you already have multiple Proxmox servers on your network.
-
@Patch I run it on my TrueNAS as the moment that sits o my 172.16.10.0/24 network... which itself feels unwanted/homeless when I have the above network issues. so moving it onto a different machine all patched into the same switch is not making sense.
Was thinking to use my previous pfSense host that has a celeron processor and patch that directly into the pfSense onto one of the open ports, thus bypassing the switch completely.
Makes some sense, meaning my unifi controller might have a better chance of surviving something... but then the something seems to happen between the pfSense and core switch, the ProMax. When it happens it, switch and the other important VM's all want to talk to the Unifi Network controller... but thats sitting on a TrueNAS which itself is now lost...issue is, trying to fix a problem i dont 100% understand the root cause of yet...
for now jsut trying to remove points of question/instability...G
-
@georgelza My thinking is that you are making a lot of settings in places that also have switching, like Proxmox and perhaps TrueNAS as well. And since you have multiple connections from some of these, there is a risk you create loops... Leave the switching to the switches and DHCP to pfsense, at least until you really see a need to set a static IP in the device itself.
Use the topology mapping in Unifi Controller as support as well. It might give you some ideas of where things may be messed up... -
@Gblenn said in Switching from igc1 -> ic0 as physical interface starts well and then collapses:
Use the topology mapping in Unifi Controller as support as well. It might give you some ideas of where things may be messed up..
issue when the wheels come off my Unifi manager is not reachable...
Only switch this is internal static assigned is the ProMax... will remove that and do a static assign on the pfSense... was actually thinking the current config, by the switch is told you are 172.16.10.2, irrespective which port is used for uplink... if anything... i would have expected more problems when the Cat6/2.5 GbE and fiber was patch / active into the pfSense... as it would not have known who is the uplink...
Will give all of this a try.
G
