Unable to reassign network port

michmoor

@stephenw10 Restored the configuration. Tried to change vlan description. Doesnt happen.

stephenw10

Mmm, I can't reproduce this either.
I would have to guess something in your config is trying to apply invalid values when you change the VLAN description. The sort of thing I could imagine it being would be and MTU or link state type that it tries to apply as an inherited value to the parent. Since you are using vlans on a lagg that is a double layer of parent interfaces.

michmoor

@stephenw10 So its definitely something in the config or at least how the GUI works with the config.xml in the background. Took a spare 6100 and loaded my config.xml and the exact same symptoms show up. I apply a vlan description change, All interfaces bounce, and the vlan change never happens.
Ive been reviewing both the <Interfaces> and <vlan? hierarchy and i couldnt tell you whats wrong. All looks good or at least nothing out of place.

Strange thing is, when i make the change directly to the config.xml, no issue. vlan change goes through and no interface flaps.

tail -f system.log when i change the vlan description the only things that stick out at me are the following

Feb  6 18:44:56 GA-FW1 arpwatch[24336]: pcap_loop: The interface disappeared
Feb  6 18:44:56 GA-FW1 kernel: lagg0.23: promiscuous mode disabled
Feb  6 18:44:56 GA-FW1 kernel: vlan5: changing name to 'lagg0.23'

Feb  6 23:44:56 GA-FW1 php-fpm[36470]: /interfaces_vlan_edit.php: Gateway, NONE AVAILABLE

Feb  6 23:44:57 GA-FW1 php-fpm[12214]: /rc.linkup: Hotplug event detected 

Feb  6 23:49:46 GA-FW1 check_reload_status[334]: rc.newwanip starting lagg0.17
Feb  6 23:49:46 GA-FW1 check_reload_status[334]: rc.newwanip starting lagg0.3

Its renewing WAN IP on my lagg facing my LAN? Something is strange.

last edit

Parsing the logs I dont know why newwanip is running on each interface and resyncing with OpenVPN.

Feb 7 00:24:33	php-fpm	69922	/rc.newwanip: Resyncing OpenVPN instances for interface DMZ.
Feb 7 00:24:30	php-fpm	8705	/rc.newwanip: Resyncing OpenVPN instances for interface SLEEPY_IOT_WLAN.
Feb 7 00:24:26	php-fpm	36470	/rc.newwanip: Resyncing OpenVPN instances for interface KIDSZONE_WLAN.
Feb 7 00:24:24	php-fpm	27432	/rc.newwanip: Resyncing OpenVPN instances for interface STORAGE.
Feb 7 00:24:21	php-fpm	90475	/rc.newwanip: Resyncing OpenVPN instances for interface WORK.
Feb 7 00:16:46	php-fpm	69922	/interfaces.php: Resyncing OpenVPN instances for interface LAN.
Feb 6 23:57:56	php-fpm	27432	/interfaces.php: Resyncing OpenVPN instances for interface WORK.
Feb 6 23:50:04	php-fpm	36470	/rc.newwanip: Resyncing OpenVPN instances for interface DMZ.
Feb 6 23:50:02	php-fpm	69922	/rc.newwanip: Resyncing OpenVPN instances for interface SLEEPY_IOT

stephenw10

To be clear you are editing the Description field in Interfaces > Assignments > VLANs > Edit?

Then saving that?

Doing that will probably push interface options up to lagg0 and to the NICs that make up lagg0. And that in turn would mean anything else using those parents would inherit it. That's probably why you see everything reload. And somewhere there something is probably trying to set an invalid value. Though I would expect to see an error logged.

Are you able to upload your config to use to examine?

michmoor

@stephenw10 I just updated my comment above adding more log messages that i find very strange.

You tell me where to upload the config and you can have it. Appreciate the help on this.

stephenw10

You can upload it here: https://nc.netgate.com/nextcloud/s/2A3mxLXwMnFEFak

Thanks

michmoor

@stephenw10 Done. Gave you the config along with my system logs during the time of the change. Just to reiterate, all this is was a vlan description change.

config.xml
systemlogs

stephenw10

Great I have that.

And just to be clear you're editing the description field in Interfaces > Assignments > VLANs > Edit?

michmoor

@stephenw10 correct.
This all started because I wanted to reassign some ports to create a lagg. My changes were never saved. Edited the .xml

Next I wanted to update vlan descriptions. Changes never saved. That’s when I did some digging and checking logs.

michmoor

Howdy @stephenw10 Did you have any free cycles to check out the items i uploaded?

stephenw10

I looked over your config and I don't see anything significant there really. I'll have to try loading it up on something and testing it when I can.

Do you know when during the logs you applied that change and it failed?

michmoor

@stephenw10 I was screen grabbing at the same time I clicked save when doing the vlan change.
So right at the top are the beginning of the flooding messages that come in.
Oddly someone is having a similar issue in the Reddit forum. Maybe it’s a NIC driver thing? That and the config seems to be the only consistent thing.
I have half a mind to install the RC tonight.

stephenw10

You have a link?

Are they also using a lagg of igc NICs?

michmoor

@stephenw10 https://www.reddit.com/r/PFSENSE/comments/10w51rk/hourly_network_drops/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

marcosm

Have you tried re-doing the configuration instead of restoring the config file? If you're able to reproduce it that way by only making related changes to a default configuration, that can help narrow down the issue.

michmoor

@marcosm There is quite a bit to restore manually.
So what i am thinking about trying is the following

re-install pfSense. Do not restore original configuration. Create VLANs and attempt to change the description. I will note the results. Afterward i will create a LAGG0. See how that goes.
Assuming the problem cannot be repeated from step 1 then i will restore my configuration. I will then blow away all vlans and lagg interfaces. recreate vlan.ids and laggs and attempt to modify.

Depending on how things go, it could very well be something funky in configuration.
The mystery is why is it that changes to interfaces and vlans through the GUI are not saved but if i edit the config.xml directly then interface changes are saved.
How does the GUI talk to the system files? I assume there is some commit check that takes place. If there is a log for that, that could reveal alot of whats going on behind the scenes.

michmoor

@marcosm

One more tidbit to kind of proved my point about the outages...

igc0 is my LAN. Not in a VLAN. Traffic not routed across the LAGG.
I change the vlan description and i have a continuous uninterrupted ping to google.com.

Reply from 172.217.13.14: bytes=32 time=6ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115
Reply from 172.217.13.14: bytes=32 time=2ms TTL=115

Ping statistics for 172.217.13.14:
    Packets: Sent = 33, Received = 33, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 17ms, Average = 6ms

Now i set up an extended ping to another VLAN that is on the LAGG. I make a vlan description change and outage..

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Request timed out.
Reply from 192.168.17.2: bytes=32 time=2ms TTL=127
Reply from 192.168.17.2: bytes=32 time=14ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127

Ping statistics for 192.168.17.2:
    Packets: Sent = 15, Received = 13, Lost = 2 (13% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 14ms, Average = 1ms

@stephenw10 i swear im not crazy :)

michmoor

Update:
Anyhthing that has to do with the LAGG triggers an outage on the LAGG.

All i did was add a VLAN tonight and the results are below.
Pings start on igc0[192.168.50.221] which is not a member of the lagg and not part of any vlan.

ping -t 192.168.17.2

Pinging 192.168.17.2 with 32 bytes of data:
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.50.254: Destination host unreachable.
Reply from 192.168.50.254: Destination host unreachable.
Request timed out.
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time<1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127
Reply from 192.168.17.2: bytes=32 time=1ms TTL=127

stephenw10

There are a few things that could be happening here:
Input validation in the GUI is preventing you making the changes because some existing setting it tries to apply at the same time is invalid. However if that were true I would expect it to throw an error in the gui when you tried to save it. And there wouldn't actually be anything applied to the interfaces so you wouldn't see the lagg bounce.
It creates a config that is invalid generating a bad config file and pfSense chooses the last valid config to use. If that was happening I would expect to see a bunch of logs indicating it.

The fact it bumps lagg implies changes are being applied to the VLAN and it's trying to propagate those to it's parent interface, lagg0.

I haven't been able to replicate it even using a vlan on a lagg of igc NICs exactly as you have.Yet.

When you save the description change do you see that shown in Diag > Backup > Config History?

michmoor

@stephenw10 Good question.

Just modified a vlan description. Change didnt stick