When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong"

bingo600

Today i wanted to do some firewall maintenance. Blocking Cloudflare tunnels on my home fwall.
https://www.bleepingcomputer.com/news/security/hackers-increasingly-abuse-cloudflare-tunnels-for-stealthy-connections/

I'm on latest PLUS

I discovered, that when i copy a rule from one IF to another IF, the copied rule appears at the top, on the new IF.
Then it seems that the Firewall: Rules - reordered firewall rules. messes with the rules and separator lines.
As in "the newly inserted copied rule) is shifting other rules down to the next separator.

That is "not nice" , as it takes a lot of time to move all the rules back "under the correct separator line".

Can anyone reproduce/verify

Last 500 General Log Entries. (Maximum 500)
Aug 8 14:58:55 	check_reload_status 	1025 	Reloading filter
Aug 8 14:58:51 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:58:51 	php-fpm 	17953 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:53:36 	check_reload_status 	1025 	Reloading filter
Aug 8 14:53:32 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:53:32 	php-fpm 	987 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:53:01 	check_reload_status 	1025 	Reloading filter
Aug 8 14:52:05 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:52:05 	php-fpm 	986 	/firewall_rules_edit.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - saved/edited a firewall rule.
Aug 8 14:51:38 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:51:38 	php-fpm 	17032 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:50:05 	check_reload_status 	1025 	Reloading filter
Aug 8 14:50:01 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:50:01 	php-fpm 	987 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:48:28 	check_reload_status 	1025 	Reloading filter
Aug 8 14:48:25 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:48:25 	php-fpm 	986 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:47:23 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:47:22 	php-fpm 	17953 	/firewall_rules_edit.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - saved/edited a firewall rule.
Aug 8 14:46:56 	check_reload_status 	1025 	Syncing firewall
Aug 8 14:46:56 	php-fpm 	987 	/firewall_rules.php: Configuration Change: admin@10.x.x.x (Local Database): Firewall: Rules - reordered firewall rules.
Aug 8 14:46:42 	check_reload_status 	1025 	Reloading filter
Aug 8 14:46:39 	check_reload_status 	1025 	Syncing firewall

IF before copying a rule from another IF

IF after copying a rule from another if

As can be seen the new rule is inserted in top here
And existing rules are shifted down , but the separator's are not.

Edit:
And ... Why is the Copy sometimes (rare) ending up at the bottom of the new IF , and mostly at the top ?
What determins if the rule ends at top or bottom ???

Edit2:
I just replicated it on my summerhouse pfS too , it's also on latest.

@stephenw10

/Bingo

stephenw10

Ok I replicated this in 23.05.1 but it appears to be fixed already in 23.09.

Upon copying the rule the separator row value is incremented:

--- /conf/backup/config-1691527722.xml	2023-08-08 21:49:50.530422000 +0100
+++ /conf/backup/config-1691527790.xml	2023-08-08 21:56:24.276501000 +0100
@@ -216,6 +216,40 @@
 		</rule>
 		<rule>
 			<id></id>
+			<tracker>1691527790</tracker>
+			<type>pass</type>
+			<interface>opt3</interface>
+			<ipprotocol>inet6</ipprotocol>
+			<tag></tag>
+			<tagged></tagged>
+			<max></max>
+			<max-src-nodes></max-src-nodes>
+			<max-src-conn></max-src-conn>
+			<max-src-states></max-src-states>
+			<statetimeout></statetimeout>
+			<statetype><![CDATA[keep state]]></statetype>
+			<os></os>
+			<srcmac></srcmac>
+			<dstmac></dstmac>
+			<source>
+				<network>lan</network>
+			</source>
+			<destination>
+				<any></any>
+			</destination>
+			<descr><![CDATA[Default allow LAN IPv6 to any rule (copied)]]></descr>
+			<bridgeto></bridgeto>
+			<updated>
+				<time>1691527790</time>
+				<username><![CDATA[admin@172.21.16.8 (Local Database)]]></username>
+			</updated>
+			<created>
+				<time>1691527790</time>
+				<username><![CDATA[admin@172.21.16.8 (Local Database)]]></username>
+			</created>
+		</rule>
+		<rule>
+			<id></id>
 			<tracker>1691527654</tracker>
 			<type>pass</type>
 			<interface>opt3</interface>
@@ -322,7 +356,7 @@
 		<separator>
 			<opt3>
 				<sep0>
-					<row>fr2</row>
+					<row>fr3</row>
 					<text><![CDATA[Block rules below here]]></text>
 					<color>bg-danger</color>
 					<if>opt3</if>
@@ -330,7 +364,8 @@
 			</opt3>
 		</separator>
 	</filter>

bingo600

@stephenw10
Thank you for your time & effort

How does Netgate suggest that I solve this issue ?

1:
Will you release a new "patch file" for 23.05-1 ?

2:
Or do i have to manually patch with a patch id ?
Is this the id to use in "Custom System Patthes"

Applied in changeset 8a12728da23fc7cb654cec4a97670ef2b6dfb239.

3:
Or do i have to upgrade to "pre" 23.09
I'd rather not if possible ....

/Bingo

bingo600

Well i'm running ZFS

I backed up my current config , and took a "Quick snapshot"

Then i applied the "patch number" to my 23.05-1 (no reboot/reroot)

If i have done it correctly (patching didn't give any errors), and the patch should function on 23.05-01 too , then it doesn't seem to fix the issue.

New copy rule, after patch is applied

IF Pre copy

..
..

IF Post copy

Maybe the fix .. fixes "rule delete" ... and not "rule insert" ???

Again thank you for looking into this.

Now to ... Either revert the patch , or restore the "snapshot"

Well ... I played safe , and restored the snapshot
Worked excellent

/Bingo

stephenw10

Yes there have been a number of patches related to that. It may be a combination of them that results in incrementing the separator row correctly.

Let me see....

bingo600

@stephenw10
Did you add/create/update a "redmine" somewhere ?
I can't see any updates in : 8a12728da23fc7cb654cec4a97670ef2b6dfb239

I gotta learn how2 make a redmine (i have only ever done one)

/Bingo

Gertjan

@bingo600 said in When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong":

(i have only ever done one

So you have an account here.
Ones logged in :

stephenw10

I didn't create a redmine report for this because it's already fixed in 23.09.

I was trying to find the combination of commits that fixes it that might be applied to 23.05.1. There are a quite a few things though.

bingo600

@stephenw10

I was trying to find the combination of commits that fixes it that might be applied to 23.05.1. There are a quite a few things though.

That would be "super handy to get" ... thanx

The 23.09 patches didn't solve it on 23.05-1

I'm in the sumerhouse right now, but i might "upgrade" my home test pfS to 23.09 to verify that it's solved.
... My "immediate" gut feeling is that the 23.09 patches should be fixing it for 23.05-1 too (if solving my issue) .. But that's purely "speculations"

Might be able to test tue/wed next week ...

/Bingo

stephenw10

@bingo600 said in When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong":

My "immediate" gut feeling is that the 23.09 patches should be fixing it for 23.05-1 too

Yes, I would expect that to be true since that's all in the run-time scripts. It's just finding exactly which commits are required to get the fix since there are many...

bingo600

@stephenw10

Yes, I would expect that to be true since that's all in the run-time scripts. It's just finding exactly which commits are required to get the fix since there are many...

So the 3..4 patches from the referred patchset isn't enough ?
I did apply that full set (the ID pulled the full patchfile)

Btw ...
Seems like 23.09 is not in my current 23.05-1 "Upgrade lists" (I only have Current (23.05-1) & Previous (23.05)) , and snapshots are offline.

I even have dug out my "cold spare" pfS here in the summerhose
Downloaded 2.7.0
And "bought" 4 new Plus keys, for my play boxes.

/Bingo

mvikman

Just adding my experience to this thread.

I also added CloudFlare Tunnel block rules on my box (23.05.1) and used the copy function to add it to the other interfaces.
All the copied rules ended up at the bottom row in every interface (all interfaces have at least 7 basic rule rows and 6 separators).

bingo600

@mvikman said in When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong":

Just adding my experience to this thread.

I also added CloudFlare Tunnel block rules on my box (23.05.1) and used the copy function to add it to the other interfaces.
All the copied rules ended up at the bottom row in every interface (all interfaces have at least 7 basic rule rows and 6 separators).

Thanx for chipping in.

I have "rarely seen" copied rules added to the bottom.
Mine end at the top line 90+ % of the time.

I have no idea what is "controlling" where they end.

/Bingo

mvikman

@bingo600
Manual page has a warning note on this btw.
Don't know if it's been there or added now.

https://docs.netgate.com/pfsense/en/latest/firewall/rule-list-intro.html#copying-firewall-rules
"When copying rules to different interfaces, they may fall at the start or the end of the target interface rule list depending on the order of the interface rules in the configuration. Be prepared to reorder the rules on the target interface before applying changes."

bingo600

@mvikman said in When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong":

@bingo600
Manual page has a warning note on this btw.
Don't know if it's been there or added now.

https://docs.netgate.com/pfsense/en/latest/firewall/rule-list-intro.html#copying-firewall-rules
"When copying rules to different interfaces, they may fall at the start or the end of the target interface rule list depending on the order of the interface rules in the configuration. Be prepared to reorder the rules on the target interface before applying changes."

I don't hope this is the "Solution" for the problem, to have to reorder all the rules.
That would be "close to" unusable ....

On some of my IF's i have 30+ rules.

If they would just add the new rule at the bottom, there would be nothing to "shift up" , and a "move" of the copied rule would be "easy".

Edit:
If the DOC has been changed (I don't think so)
Then it reminds me of the "Old IBM MVS Mainframe days" ...
We had a particular nasty BUG , that kept popping up .. In the end IBM just changed the DOC, to "don't do that" .... - Issue solved

/Bingo

stephenw10

Since the rule ordering is all-important in pfSense there will likely never be a perfect solution here. Keeping the separators in the correct place is relatively easy once you know where the new rule will go.
If all your interfaces have similar rules I would expect it to work. Otherwise adding the rule at the end of the table is probably safest.

mvikman

I would say that best might be that copied rule is always added as the last row as a disabled rule, then you can move it to correct position and enable it without accidentally compromising your rule set.

bingo600

@marcosm
Saw you were active in : https://forum.netgate.com/post/1122028

Regarding my rule copy/duplicate to another IF issue, in this thread.

Would : https://redmine.pfsense.org/issues/14691
Aka Patch: https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/26b97b650457ba98360b5648dd801fd0adb567a5

Fix my issue on 23.05.1 ?
Can i apply it there ?

What about : https://redmine.pfsense.org/issues/14619
Patch: https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/8a12728da23fc7cb654cec4a97670ef2b6dfb239

Does that have to be applied first ??

I tried to make a snapshot , and apply : https://redmine.pfsense.org/issues/14619
That didn't solve my copy issue, so i reverted.

But will the combo of the two do it ??

Thank you for your work

/Bingo

marcosm

@bingo600 There are a few patches that would be needed in order to fully fix it. I'll see about adding this to the System Patches package. For now, here's the patch for CE/Plus with all of the fixes.
14619_14691_plus.patch
14619_14691_ce.patch

bingo600

@marcosm said in When copying a rule from one if to another, it seems like pfSense is reordering the rules "wrong":

@bingo600 There are a few patches that would be needed in order to fully fix it. I'll see about adding this to the System Patches package. For now, here's the patch for CE/Plus with all of the fixes.
14619_14691_plus.patch
14619_14691_ce.patch

I made a snapshot , applied the plus patch and "Rerooted" the Box.

Then i copied a rule from one IF to another , and the Rule ended up at the bottom
That's fine by me THANK YOU

Btw:
Is there any "Easy" explanation to when a copied rule eds up at the top or the bottom ??
On my box it seems a little random.

Thank you for your support

/Bingo