Unable to reassign network port

[stephenw10]

Hmm, and that also fails?

What does the config diff show if you just try to change the description of an existing VLAN?

[michmoor]

@stephenw10 updated the screen shot. Wrote a test description. You see a config change but nothing in the GUI.

[stephenw10]

But that's a previous config change? It doesn't include the VLAN changes.

Or is that timestamp when you actually made the change?

[michmoor]

@stephenw10 I just did a vlan change. 11:16

[stephenw10]

Hmm, what was the actual change you made? What was the new description you tried to set?

[michmoor]

@stephenw10 just wrote the word 'Test' I wish i could show you this in real time. I also went to Services / Auto Configuration Backup/ Revision Information
Look at the vlan hierarchy and its unchanged..

[michmoor]

Some good news.
As a first step i decided to delete all sub-interfaces , vlans and lagg0. All that is left is WAN and LAN.
First i re-created the lagg0.
Secondly, i re-created my vlan tags and assigned them all to parent interface lagg0.
I changed vlan descriptions multiple times and each time the change is reflected in the GUI.

Because the interfaces were deleted and i had to create them again, attempting to restore Firewall rules from backup config i receive the following error message

In the back of my mind, i had a suspicion that somehow the interface mappings were messed up somehow. That looks to be the case for sure.

Fatal error: Uncaught Exception: XML error: SSHDATA at line 10148 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #13, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #13, 'mldata>\n\t\t

My other question. Is it normal for the LAGG to flap when changing the description of a VLAN tag? The LED lights on the NetGate do go off and come back on. There is ping loss and the system logs do see an UP/DOWN event so im not making it up.

[stephenw10]

That's a known issue: https://redmine.pfsense.org/issues/13132

Looks like restoring a partial config created two sshdata sections. You should be able to manually remove one.

When you save the VLAN config to change the description is recreates the VLAN and doing that attempts to apply values to/from the parent interfaces. So, yes, I might expect to see the lagg bounce. You might not see the NICs re-link if nothing gets reconfigured.

Steve

[michmoor]

@stephenw10 Alrighty now i am very confident i know what the problem is. The issue is with the LAGG itself.
Create a LAGG, assign your vlans and attempt to make modifications. That will leave you with an outage.
I now moved vlans over to igc2 [really any interface] and no issue.
Move it back to the LAGG, and then the interface bounce, no vlan description changes. Its a total outage.
I have managed to reproduce this so far on two devices, specifically 6100s.
This explains why I can throw my config.xml on another 6100 and the problem follows. It's the LAGG not playing nice somehow. Driver related? Unsure.

Not trying to throw shade here but i think thats why this wasnt tested at all even though my redmine and TAC case was closed with 'unable to reproduce' message. The testing methodology isnt good even though i did point out that the LAGG is part of the implementation.
Throw the interfaces in a LAGG and my suspicion is that you will see the trouble I'm seeing with VLAN changes.

[stephenw10]

I'm testing using a VLAN on a lagg of igc NICs on a 6100. And I can't reproduce it.
Specifically:

	<vlans>
		<vlan>
			<if>lagg0</if>
			<tag>100</tag>
			<pcp></pcp>
			<descr><![CDATA[Not test]]></descr>
			<vlanif>lagg0.100</vlanif>
		</vlan>
	</vlans>
	<laggs>
		<lagg>
			<members>igc2,igc3</members>
			<descr><![CDATA[lagg0]]></descr>
			<laggif>lagg0</laggif>
			<proto>lacp</proto>
			<lacptimeout>slow</lacptimeout>
			<lagghash>l2,l3,l4</lagghash>
		</lagg>
	</laggs>

I agree the lagg likely is important there. And if it is some property that can't be applied to the lagg members the NIC type would also be important.

I did also consider whether having the lagg interface itself assigned would make a difference since the first test I ran did have that and you don't. But unassigning it did not change anything. You might try assigning it to see if that changes anything for you.

I'll test your config dircetly and see if that triggers anything ....

[michmoor]

@stephenw10 If this really cant be replicated on your end then i honestly dont know.
Any changes related to a LAGG such as interface descriptions shouldnt bounce a LAGG, at least in practice. Is this what you are seeing on your end? Is part of my problem description accurate on that?

[stephenw10]

Ok, first I have to apologise because I failed to notice your config had spoofed the WAN MAC. So for a very brief period after I uploaded it onto a 6100 here the block rule I had added was for the wrong IP and it had outbound access. So I hope it didn't cause you any issues if it updated your dyndns with my IP for example.

But, having corrected that, I'm still unable to replicate what you're seeing. I changed the description of VLAN 17 and it applied as expected:
login-to-view

--- /conf/backup/config-1675903864.xml	2023-02-08 20:10:42.339035000 -0500
+++ /conf/config.xml	2023-02-08 20:10:42.358714000 -0500
@@ -4274,7 +4274,7 @@
 			<if>lagg0</if>
 			<tag>17</tag>
 			<pcp></pcp>
-			<descr><![CDATA[Test]]></descr>
+			<descr><![CDATA[Test2]]></descr>
 			<vlanif>lagg0.17</vlanif>
 		</vlan>
 		<vlan>
@@ -4294,9 +4294,9 @@
 	</vlans>
 	<qinqs></qinqs>
 	<revision>
-		<time>1675903864</time>
-		<description><![CDATA[admin@192.168.50.200 (Local Database Fallback): VLAN interface added]]></description>
-		<username><![CDATA[admin@192.168.50.200 (Local Database Fallback)]]></username>
+		<time>1675905042</time>
+		<description><![CDATA[admin@172.21.16.8 (Local Database Fallback): VLAN interface added]]></description>
+		<username><![CDATA[admin@172.21.16.8 (Local Database Fallback)]]></username>
 	</revision>
 	<gateways>
 		<defaultgw4>WAN_DHCP</defaultgw4>

However, because I have blocked the firewall access to an upstream connection there are several differences.
It is unable to access packages so none are installed and you have a lot of packages in that config.
It cannot access ACB and the ruleset takes a very long time to load, like 5mins, because of the unpopulated aliases from pfBlocker.

I suspect what you're seeing is somehow caused by interaction with one or more packages you have. I'd have to guess Wireguard or pfBlocker as they both produce config changes when they see interface changes.

Are you able to test disabling or removing those?

[michmoor]

@stephenw10 said in Unable to reassign network port:

Ok, first I have to apologise because I failed to notice your config had spoofed the WAN MAC. So for a very brief period after I uploaded it onto a 6100 here the block rule I had added was for the wrong IP and it had outbound access. So I hope it didn't cause you any issues if it updated your dyndns with my IP for example.

That was you?!?!? Just kidding i didnt notice any outage on my end

Im going to disable pfBlocker first. Bringing down the WireGuard tunnels i lose access to my VPC.
I'll test and let you know.

[michmoor]

@michmoor Disabled both packages and tested. No change.
Another thing i have found through my testing. If i disable the interface. Then go to the vlan and change the description, the changes stick in the GUI.

[michmoor]

I suppose we should close this out here and these are my conclusions. Just to note this can be replicated to another 6100 but others cannot replicate it. So this issue varies between devices.

1. VLAN description changes cannot happen while the Interface is enabled. I first have to disable the interface, go back to the VLAN assignments and change the description there. Enable the interface again.

2. Any changes to the LAGG [adding a vlan, deleting a vlan, changing description] causes the LAGG to bounce. Any connectivity riding that LAGG will experience an outage. It shouldn't be like this as other vendors do not function like this so this really should be looked at. Its possible for other vendors to write proprietary code for LAGGs to avoid this issue?

3. Late last night i discovered that if you do not use a LAG but instead use a single interface trunked. any changes made to that trunk interface will result in a link bounce. Again, it shouldnt be this way but i managed to replicate it on another device.

Overall, using trunked interfaces on pf is very risky in a production environment as any changes to it result in a momentary blip. I am curious if this would happen to me on a 4100 as that has different NIC drivers i believe. I still strongly suspect the NIC is the issue.

[stephenw10]

The 4100 uses identical NICs to the 6100 (and 8200).

The config not applying has to be a result of one the packages you have installed. If you removed all the packages you would be at the same point I was testing from (almost) where it does not happen.

Steve

[michmoor]

@stephenw10 I'm really beginning to dislike the fact you are always right

I removed pfblockerNG
login-to-view

I then went in to change the vlan description and wouldnt you know it.......Applied right away.

There is still an outage tho.
login-to-view

So this leaves me with two questions

1. where do we go from here? Do i reach out to the package maintener to describe the problem and see if theres a fix?
2. Is the port flapping just what happens and thats the nature of how lagg is implemented?

[stephenw10]

Ah, that's interesting...hmm let me think about that. It could be simply the time it takes to reload everything some how. I assume you do not see it take 5mins to reload the filter when pfBlocker is populated correctly?

Applying changes to a VLAN on the LAGG is likely to bounce the LAGG because it reapplies the interfaces and the properties are inherited. I'm not sure if that can be prevented entirely.

[michmoor]

@stephenw10 does it take a long time for pfblocker to update you mean? No it’s quick. Maybe a minute or 2 at the most.
Can you load pfblocker on your test device and replicate?

[stephenw10]

No I mean the filter load so if you go to Status > Filter Reload and initiate the load how long does it take? It should be seconds.