Unable to reassign network port

[michmoor]

Re-install went through without an issue. Couldnt do the recovery of the config.xml as the system stated it couldnt find it. No matter thats what 3-2-1 backup strategies are for.

Nevertheless, this didnt fix the issue. It is the most bizarre thing.

So whats my next steps? Any ideas?

[stephenw10]

So you restored the config and then couldn't change the descriptions? Or reconfigured it and then couldn't?

[michmoor]

@stephenw10 Restored the configuration. Tried to change vlan description. Doesnt happen.

[stephenw10]

Mmm, I can't reproduce this either.
I would have to guess something in your config is trying to apply invalid values when you change the VLAN description. The sort of thing I could imagine it being would be and MTU or link state type that it tries to apply as an inherited value to the parent. Since you are using vlans on a lagg that is a double layer of parent interfaces.

[michmoor]

@stephenw10 So its definitely something in the config or at least how the GUI works with the config.xml in the background. Took a spare 6100 and loaded my config.xml and the exact same symptoms show up. I apply a vlan description change, All interfaces bounce, and the vlan change never happens.
Ive been reviewing both the <Interfaces> and <vlan? hierarchy and i couldnt tell you whats wrong. All looks good or at least nothing out of place.

Strange thing is, when i make the change directly to the config.xml, no issue. vlan change goes through and no interface flaps.

tail -f system.log when i change the vlan description the only things that stick out at me are the following

Feb  6 18:44:56 GA-FW1 arpwatch[24336]: pcap_loop: The interface disappeared
Feb  6 18:44:56 GA-FW1 kernel: lagg0.23: promiscuous mode disabled
Feb  6 18:44:56 GA-FW1 kernel: vlan5: changing name to 'lagg0.23'

Feb  6 23:44:56 GA-FW1 php-fpm[36470]: /interfaces_vlan_edit.php: Gateway, NONE AVAILABLE

Feb  6 23:44:57 GA-FW1 php-fpm[12214]: /rc.linkup: Hotplug event detected 

Feb  6 23:49:46 GA-FW1 check_reload_status[334]: rc.newwanip starting lagg0.17
Feb  6 23:49:46 GA-FW1 check_reload_status[334]: rc.newwanip starting lagg0.3

Its renewing WAN IP on my lagg facing my LAN? Something is strange.

last edit

Parsing the logs I dont know why newwanip is running on each interface and resyncing with OpenVPN.

Feb 7 00:24:33	php-fpm	69922	/rc.newwanip: Resyncing OpenVPN instances for interface DMZ.
Feb 7 00:24:30	php-fpm	8705	/rc.newwanip: Resyncing OpenVPN instances for interface SLEEPY_IOT_WLAN.
Feb 7 00:24:26	php-fpm	36470	/rc.newwanip: Resyncing OpenVPN instances for interface KIDSZONE_WLAN.
Feb 7 00:24:24	php-fpm	27432	/rc.newwanip: Resyncing OpenVPN instances for interface STORAGE.
Feb 7 00:24:21	php-fpm	90475	/rc.newwanip: Resyncing OpenVPN instances for interface WORK.
Feb 7 00:16:46	php-fpm	69922	/interfaces.php: Resyncing OpenVPN instances for interface LAN.
Feb 6 23:57:56	php-fpm	27432	/interfaces.php: Resyncing OpenVPN instances for interface WORK.
Feb 6 23:50:04	php-fpm	36470	/rc.newwanip: Resyncing OpenVPN instances for interface DMZ.
Feb 6 23:50:02	php-fpm	69922	/rc.newwanip: Resyncing OpenVPN instances for interface SLEEPY_IOT

[stephenw10]

To be clear you are editing the Description field in Interfaces > Assignments > VLANs > Edit?

Then saving that?

Doing that will probably push interface options up to lagg0 and to the NICs that make up lagg0. And that in turn would mean anything else using those parents would inherit it. That's probably why you see everything reload. And somewhere there something is probably trying to set an invalid value. Though I would expect to see an error logged.

Are you able to upload your config to use to examine?

[michmoor]

@stephenw10 I just updated my comment above adding more log messages that i find very strange.

You tell me where to upload the config and you can have it. Appreciate the help on this.

[stephenw10]

You can upload it here: https://nc.netgate.com/nextcloud/s/2A3mxLXwMnFEFak

Thanks

[michmoor]

@stephenw10 Done. Gave you the config along with my system logs during the time of the change. Just to reiterate, all this is was a vlan description change.

config.xml
systemlogs

[stephenw10]

Great I have that.

And just to be clear you're editing the description field in Interfaces > Assignments > VLANs > Edit?

[michmoor]

@stephenw10 correct.
This all started because I wanted to reassign some ports to create a lagg. My changes were never saved. Edited the .xml

Next I wanted to update vlan descriptions. Changes never saved. That’s when I did some digging and checking logs.

[michmoor]

Howdy @stephenw10 Did you have any free cycles to check out the items i uploaded?

[stephenw10]

I looked over your config and I don't see anything significant there really. I'll have to try loading it up on something and testing it when I can.

Do you know when during the logs you applied that change and it failed?

[michmoor]

@stephenw10 I was screen grabbing at the same time I clicked save when doing the vlan change.
So right at the top are the beginning of the flooding messages that come in.
Oddly someone is having a similar issue in the Reddit forum. Maybe it’s a NIC driver thing? That and the config seems to be the only consistent thing.
I have half a mind to install the RC tonight.

[stephenw10]

You have a link?

Are they also using a lagg of igc NICs?

[michmoor]

@stephenw10 https://www.reddit.com/r/PFSENSE/comments/10w51rk/hourly_network_drops/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

[marcosm]

Have you tried re-doing the configuration instead of restoring the config file? If you're able to reproduce it that way by only making related changes to a default configuration, that can help narrow down the issue.

[michmoor]

@marcosm There is quite a bit to restore manually.
So what i am thinking about trying is the following

1. re-install pfSense. Do not restore original configuration. Create VLANs and attempt to change the description. I will note the results. Afterward i will create a LAGG0. See how that goes.

2. Assuming the problem cannot be repeated from step 1 then i will restore my configuration. I will then blow away all vlans and lagg interfaces. recreate vlan.ids and laggs and attempt to modify.

Depending on how things go, it could very well be something funky in configuration.
The mystery is why is it that changes to interfaces and vlans through the GUI are not saved but if i edit the config.xml directly then interface changes are saved.
How does the GUI talk to the system files? I assume there is some commit check that takes place. If there is a log for that, that could reveal alot of whats going on behind the scenes.

[michmoor]

@marcosm

One more tidbit to kind of proved my point about the outages...

igc0 is my LAN. Not in a VLAN. Traffic not routed across the LAGG.
I change the vlan description and i have a continuous uninterrupted ping to google.com.

Reply from 172.217.13.14: bytes=32 time=6ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115

Ping statistics for 172.217.13.14:
    Packets: Sent = 33, Received = 33, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 17ms, Average = 6ms

Now i set up an extended ping to another VLAN that is on the LAGG. I make a vlan description change and outage..

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Request timed out.
Reply from 192.168.17.2: bytes=32 time=2ms TTL=127
Reply from 192.168.17.2: bytes=32 time=14ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127

Ping statistics for 192.168.17.2:
    Packets: Sent = 15, Received = 13, Lost = 2 (13% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 14ms, Average = 1ms

@stephenw10 i swear im not crazy :)

[michmoor]

Update:
Anyhthing that has to do with the LAGG triggers an outage on the LAGG.

All i did was add a VLAN tonight and the results are below.
Pings start on igc0[192.168.50.221] which is not a member of the lagg and not part of any vlan.

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127