Limiter config disappered

[michmoor]

Ive had very strange problems dealing with limiters in the past and the strangness continues. The symptoms I have after a configured Limiter and applied to a firewall rule is that after a few days or maybe a week, the Internet stops working. Further investigation shows that DNS stops working. Inter-vlan routing is still fine if i try to access resources using IP instead of DNS.

Another symptom is during troubleshooting if i see what is taking up the most CPU resources its always 'if_io_tgg_2' which is traffic related. See below.
🔒 Log in to view

Right away once i saw all of these issues i knew this was related to my Limiter i configured a week ago. So the first step is just delete the limiter. To my surprise.....The configuration is gone..

🔒 Log in to view

This is extremely odd. When i checked the firewall rule it is applied to it shows that its there.
I check the config.xml file and its there.

🔒 Log in to view

Santiy check of my configuration history shows that no deletions or modifications were done to this limiter after its applied.

Limiters are causing me a lot of grief but these situations i can reproduce but the problem is that its sporadic when it happens and it takes down the entire network.

The solution to when the network gets in this state is to pull the power on the firewall. Performing a reboot through the GUI doesn't do anything.

[michmoor]

This seems close to my issue. Only different is that i set mine to 50Mbps as you can see from the previous screen shot.

https://redmine.pfsense.org/issues/15982

<dnshaper>
		<queue>
			<guestnet-upload-50mb>
				<name>GuestNet-Upload-50Mb</name>
				<number>2</number>
				<qlimit></qlimit>
				<plr></plr>
				<description></description>
				<bandwidth>
					<item>
						<bw>50</bw>
						<burst></burst>
						<bwscale>Mb</bwscale>
						<bwsched>none</bwsched>
					</item>
				</bandwidth>
				<enabled>on</enabled>
				<buckets></buckets>
				<mask>none</mask>
				<maskbits></maskbits>
				<maskbitsv6></maskbitsv6>
				<delay>0</delay>
				<sched>wf2q+</sched>
				<aqm>droptail</aqm>
				<ecn></ecn>
			</guestnet-upload-50mb>
			<guestnet-download-50mb>
				<name>GuestNet-Download-50Mb</name>
				<number>1</number>
				<qlimit></qlimit>
				<plr></plr>
				<description></description>
				<bandwidth>
					<item>
						<bw>50</bw>
						<burst></burst>
						<bwscale>Mb</bwscale>
						<bwsched>none</bwsched>
					</item>
				</bandwidth>
				<enabled>on</enabled>
				<buckets></buckets>
				<mask>none</mask>
				<maskbits></maskbits>
				<maskbitsv6></maskbitsv6>
				<delay>0</delay>
				<sched>wf2q+</sched>
				<aqm>droptail</aqm>
				<ecn></ecn>
			</guestnet-download-50mb>
		</queue>

[michmoor]

Adding one more bit.

When the limiter malfunctions and the network becomes unstable, CPU is through the roof.
This is the exact same condition that happened last year when i was trying to troubleshoot this problem. I later figured out it was due to a configured limiter. Once I deleted it all my instability went away.
Good thing i remembered because those symptoms are back...
Solution: Delete the limiter all together. But when i went to go do that, its no longer in the GUI.

🔒 Log in to view

🔒 Log in to view

[stephenw10]

Hmm, so the config file itself never changed?

Do you have MIM enabled on that system?

Do you see anything in Diag > Limiter Info?

[michmoor]

@stephenw10
Config file never changed.
I do have MIM enabled.
Diag > Limiter says i have no limiters configured. Which is crazy! Even my firewall rule using the Limiter stated it was there (gear icon denoting the advanced options being used). When i went into the firewall rule, the limiter In/Out configuration was set to None.
Its like it never happened but for sure the limiter was what was causing my connectivity issue.

I think i am hitting that redmine bug about the disappearing configuration but why at 50Mbps..?

[stephenw10]

Doers it return if you reload the ruleset in Status > Filter Reload?

Does it return if you reboot?

[michmoor]

@stephenw10

Performing a Filter Reload

🔒 Log in to view

Sadly, the Limiters configuration is not present in the GUI.

I don't want to restart the firewall now that its working but i don't think it will help in this case (could be wrong tho)

[stephenw10]

Do they exist in /tmp/config.cache?

[michmoor]

@stephenw10 Yes sir

🔒 Log in to view

[provels]

Just spitballing, but could it be a browser/browser cache issue? Try another browser, another machine?

[michmoor]

@provels nothing is off the table.

I just tried FF and the problem is still there.

Hate to say it again but this looks like that redmine bug i noted above. Limiter just disappears after a reboot. The only difference is that the OP was setting a high limit (~4Gbps) while i am using 50Mbps

[stephenw10]

I would think that rebooting to make sure the Limiters are loaded from the config as expected should be the next step if you can.

[marcosm]

Thanks for the report. I've opened a redmine for this here https://redmine.pfsense.org/issues/16051 and will post a workaround there later.

[michmoor]

@marcosm dont think that redmine is relevant here, no?

edit:
You mean this one. https://redmine.pfsense.org/issues/16051

So its something to do with MIM enabled. Very interesting. Is there a commitID i can try to test?

[marcosm]

@michmoor Not yet - I need to work on it further on Monday.

[michmoor]

@marcosm no problem. Enjoy the weekend.

[marcosm]

I've updated the redmine with additional info that you can test out if you'd like.

[michmoor]

@marcosm

Confirmed that once i applied the PHP code the limiters came back (no reboot needed) and can be applied.

Got this notice as well.

🔒 Log in to view

If i reboot my firewall again, do i have to re-apply this PHP code?

25.03 i take it has the perm fix.

[stephenw10]

No, patches survive a reboot. They may not survive an update but, yes, this would be in 25.03 anyway so you shouldn't need to do anything.