Unable to reassign network port

stephenw10

I looked over your config and I don't see anything significant there really. I'll have to try loading it up on something and testing it when I can.

Do you know when during the logs you applied that change and it failed?

michmoor

@stephenw10 I was screen grabbing at the same time I clicked save when doing the vlan change.
So right at the top are the beginning of the flooding messages that come in.
Oddly someone is having a similar issue in the Reddit forum. Maybe it’s a NIC driver thing? That and the config seems to be the only consistent thing.
I have half a mind to install the RC tonight.

stephenw10

You have a link?

Are they also using a lagg of igc NICs?

michmoor

@stephenw10 https://www.reddit.com/r/PFSENSE/comments/10w51rk/hourly_network_drops/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

marcosm

Have you tried re-doing the configuration instead of restoring the config file? If you're able to reproduce it that way by only making related changes to a default configuration, that can help narrow down the issue.

michmoor

@marcosm There is quite a bit to restore manually.
So what i am thinking about trying is the following

re-install pfSense. Do not restore original configuration. Create VLANs and attempt to change the description. I will note the results. Afterward i will create a LAGG0. See how that goes.
Assuming the problem cannot be repeated from step 1 then i will restore my configuration. I will then blow away all vlans and lagg interfaces. recreate vlan.ids and laggs and attempt to modify.

Depending on how things go, it could very well be something funky in configuration.
The mystery is why is it that changes to interfaces and vlans through the GUI are not saved but if i edit the config.xml directly then interface changes are saved.
How does the GUI talk to the system files? I assume there is some commit check that takes place. If there is a log for that, that could reveal alot of whats going on behind the scenes.

michmoor

@marcosm

One more tidbit to kind of proved my point about the outages...

igc0 is my LAN. Not in a VLAN. Traffic not routed across the LAGG.
I change the vlan description and i have a continuous uninterrupted ping to google.com.

Reply from 172.217.13.14: bytes=32 time=6ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115

Ping statistics for 172.217.13.14:
    Packets: Sent = 33, Received = 33, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 17ms, Average = 6ms

Now i set up an extended ping to another VLAN that is on the LAGG. I make a vlan description change and outage..

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Request timed out.
Reply from 192.168.17.2: bytes=32 time=2ms TTL=127
Reply from 192.168.17.2: bytes=32 time=14ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127

Ping statistics for 192.168.17.2:
    Packets: Sent = 15, Received = 13, Lost = 2 (13% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 14ms, Average = 1ms

@stephenw10 i swear im not crazy :)

michmoor

Update:
Anyhthing that has to do with the LAGG triggers an outage on the LAGG.

All i did was add a VLAN tonight and the results are below.
Pings start on igc0[192.168.50.221] which is not a member of the lagg and not part of any vlan.

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127

stephenw10

There are a few things that could be happening here:
Input validation in the GUI is preventing you making the changes because some existing setting it tries to apply at the same time is invalid. However if that were true I would expect it to throw an error in the gui when you tried to save it. And there wouldn't actually be anything applied to the interfaces so you wouldn't see the lagg bounce.
It creates a config that is invalid generating a bad config file and pfSense chooses the last valid config to use. If that was happening I would expect to see a bunch of logs indicating it.

The fact it bumps lagg implies changes are being applied to the VLAN and it's trying to propagate those to it's parent interface, lagg0.

I haven't been able to replicate it even using a vlan on a lagg of igc NICs exactly as you have.Yet.

When you save the description change do you see that shown in Diag > Backup > Config History?

michmoor

@stephenw10 Good question.

Just modified a vlan description. Change didnt stick

stephenw10

Hmm, and that also fails?

What does the config diff show if you just try to change the description of an existing VLAN?

michmoor

@stephenw10 updated the screen shot. Wrote a test description. You see a config change but nothing in the GUI.

stephenw10

But that's a previous config change? It doesn't include the VLAN changes.

Or is that timestamp when you actually made the change?

michmoor

@stephenw10 I just did a vlan change. 11:16

stephenw10

Hmm, what was the actual change you made? What was the new description you tried to set?

michmoor

@stephenw10 just wrote the word 'Test' I wish i could show you this in real time. I also went to Services / Auto Configuration Backup/ Revision Information
Look at the vlan hierarchy and its unchanged..

michmoor

Some good news.
As a first step i decided to delete all sub-interfaces , vlans and lagg0. All that is left is WAN and LAN.
First i re-created the lagg0.
Secondly, i re-created my vlan tags and assigned them all to parent interface lagg0.
I changed vlan descriptions multiple times and each time the change is reflected in the GUI.

Because the interfaces were deleted and i had to create them again, attempting to restore Firewall rules from backup config i receive the following error message

In the back of my mind, i had a suspicion that somehow the interface mappings were messed up somehow. That looks to be the case for sure.

Fatal error: Uncaught Exception: XML error: SSHDATA at line 10148 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #13, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #13, 'mldata>\n\t\t

My other question. Is it normal for the LAGG to flap when changing the description of a VLAN tag? The LED lights on the NetGate do go off and come back on. There is ping loss and the system logs do see an UP/DOWN event so im not making it up.

stephenw10

That's a known issue: https://redmine.pfsense.org/issues/13132

Looks like restoring a partial config created two sshdata sections. You should be able to manually remove one.

When you save the VLAN config to change the description is recreates the VLAN and doing that attempts to apply values to/from the parent interfaces. So, yes, I might expect to see the lagg bounce. You might not see the NICs re-link if nothing gets reconfigured.

Steve

michmoor

@stephenw10 Alrighty now i am very confident i know what the problem is. The issue is with the LAGG itself.
Create a LAGG, assign your vlans and attempt to make modifications. That will leave you with an outage.
I now moved vlans over to igc2 [really any interface] and no issue.
Move it back to the LAGG, and then the interface bounce, no vlan description changes. Its a total outage.
I have managed to reproduce this so far on two devices, specifically 6100s.
This explains why I can throw my config.xml on another 6100 and the problem follows. It's the LAGG not playing nice somehow. Driver related? Unsure.

Not trying to throw shade here but i think thats why this wasnt tested at all even though my redmine and TAC case was closed with 'unable to reproduce' message. The testing methodology isnt good even though i did point out that the LAGG is part of the implementation.
Throw the interfaces in a LAGG and my suspicion is that you will see the trouble I'm seeing with VLAN changes.

stephenw10

I'm testing using a VLAN on a lagg of igc NICs on a 6100. And I can't reproduce it.
Specifically:

	<vlans>
		<vlan>
			<if>lagg0</if>
			<tag>100</tag>
			<pcp></pcp>
			<descr><![CDATA[Not test]]></descr>
			<vlanif>lagg0.100</vlanif>
		</vlan>
	</vlans>
	<laggs>
		<lagg>
			<members>igc2,igc3</members>
			<descr><![CDATA[lagg0]]></descr>
			<laggif>lagg0</laggif>
			<proto>lacp</proto>
			<lacptimeout>slow</lacptimeout>
			<lagghash>l2,l3,l4</lagghash>
		</lagg>
	</laggs>

I agree the lagg likely is important there. And if it is some property that can't be applied to the lagg members the NIC type would also be important.

I did also consider whether having the lagg interface itself assigned would make a difference since the first test I ran did have that and you don't. But unassigning it did not change anything. You might try assigning it to see if that changes anything for you.

I'll test your config dircetly and see if that triggers anything ....