SG-5100 UDP States still alive after schedule expires. [2.4.5-RELEASE (amd64)]
-
All,
I am running an SG-5100 on 2.4.5-RELEASE (amd64) with Dual WAN. I love the setup, but I do have one last issue I am trying to iron out.
I have a rule with a schedule for my LAN nets to allow internet access. Followed immediately by a rule to block all to the same alias. This alias is for (Computers, Tablets, & Game Consoles). After the schedule expires, this group, as expected, cannot open any new pages or games.
However, when the schedule expires it shows no states under the scheduled rule, but if I go to the Status/States page and filter for states for the test machine (192.168.0.5) in that group which running a Roblox game it is still humming along. In fact we tested and it will run all night long.
From this thread: UDP Traffic not blocked after schedule expires - I used the commands expounded by Derelict to determine which states are still alive and the rules that are letting it through.
I will agree with Derelict, that if they are getting through some rule is allowing it.
To that end I am posting my firewall rules, NAT rules, States file showing still active states on 192.168.0.5 after the schedule expired with the two UDP states highlighted in the States file that are allowing Roblox to continue to work [Roblox UDP connections highlighted with: <<<<<<<<< Roblox >>>>>>>>>]. I know the state under rule 64 is responsible, because if I kill it, the Roblox game shows that it got disconnected.
NOTE: I have Redacted all files to remove any ISP IP information (my IPs, ISP gateways and IP ranges), but nothing else. I don't care if you know my private IP settings and ranges.I also have a file with the information about the two rules that are allowing the traffic to continue. These are rules 63 & 64. I don't know how to interpret all the information shown on the verbose responses of the GREP commands.
These two rules (63 & 64) and one other (65) appear to be "hidden rules" since I cannot find any of these rules on any page of the GUI.
I also have the System/Advanced/Misc rule set to kill states when schedules expire [i.e. option NOT set]. I have attached an image showing this.
What I am looking for is why this happens. I only have one rule that allows this machine to get out to the internet by a schedule. When it expires, I expect ALL states from that machine to be unceremoniously KILLED.
If it is from another rule, where can I edit the rule?
Please inform me what I need to make this work as expected, or explain to me why it doesn't work!
Here are the files:
FW-Rules-Redacted.txt
NAT Rules-Redacted.txt
192.168.0.5 Host States-Redacted.txt [after schedule has expired]
192.168.0.5 Rules-Redacted.txt
[edited to only show the "Schedules" setting in this section]I could create a Cron to kill it all for this Alias group of machines, but this in my estimation presents a security breach, since communication through the firewall is allowed without a clearly editable rule. It also defeats the purpose of being able to enable a single rule to override the schedule for "special occasions" - i.e. friends over or a holiday.
Thanks for reading!
Phizix
-
All,
Further investigation last night shows that before the schedule expires, the outbound connection is generated by rule 98 - "USER_RULE: Allow Internet by Schedule"
And after the Schedule expires, the original outbound connection is deleted, but is automagically recreated by rule 63 - "let out anything IPv4 from firewall host itself"
Here is a file showing the states and rules before and after the Schedule expires:
States - Before and After-Redacted.txtTo confirm, after the Schedule expires this evening, I plan to try deleting the state created under rule 63 to see if it keeps recreating it.
I have to consider this either a bug or a design flaw. After the schedule expires and the state created by rule 98 is deleted, it is very bad form for the firewall to automagically recreate the state by rule 63. At least in my opinion.
Phizix
-
@jimp or other pfSense programmers,
I have looked through the issues on Redmine and found Bug #9615 [Connections permitted by a schedule are not killed when that schedule expires]. This appears to be the same as this bug I have encountered. The information I have supplied in this forum thread seems to be more complete than Victor's information.
Accordingly I created an account and added an update to the issue.
@jimp the reason I put your name here is because your name is associated with a note in this bug. :-) It does not appear to be assigned.
It is not just the UDP connections, but a number of TCP connections that also get reconnected by rule 63 (or it's equivalent on another system). I just focused on the UDP because it was easier to track down since there were only a couple of UDP connections.
Phizix
-
All,
I continue to investigate this mystery. It appears related to the automatic NAT rules generated by the system.
I have figured out why using "pfctl -k <internal-ip>/32 ; pfctl -k 0.0.0.0/0 -k <internal-ip>/32" does not kill a connection. For example on my internal network my machine is 192.168.0.100. So if I use "pfctl -k 192.168.0.100/32 ; pfctl -k 0.0.0.0/0 -k 192.168.0.100/32" this does kill all states with my machine as the source and as the destination.
The problem is that the intermediate Gateway IP map to the external site is NOT deleted by that command. The example previously used on our test machine 192.168.0.5 in the file shown previously (UDP States - Before and After-Redacted.txt) shows that the state mapped from the WAN interface [Spectrum in this case] is created on what I assume is the auto-generated NAT rule (#64) survives the end of schedule.
I experimented using my machine and killed all states from my internal ip address to a particular internet address and from it to my internal ip.
Then I checked states to that internet address, and the state from the Firewall WAN address to it survives. Furthermore, it automatically creates a new state based on the information in the surviving state to allow anything from that internet address back to my machine by creating a state under "let out anything IPv4 from firewall host itself".
I noticed that the arrow in the new state points the opposite direction from the state generated under the user rule. In other words the user rule generated state was the internet address/port from my local ip/port, but the new state generated by the "let out anything IPv4 from firewall host itself" rule shows the internet address/port to my local ip/port.
I would have assumed that this stranded port would fade away into oblivion, but apparently not.
In order to actually kill the connection, the state from the WAN ip/port to the internet address/port has to also be killed. As in this state example:
The only way I have found then, to make schedules work, is to use Cron with "pfctl -F state" a minute or two after a schedule expires.
Phizix
-
SO lastly, the question is, is there an easy way to kill states by their "original source"? As shown in parenthesis in the image in the post I made previously -- (192.168.0.5:random).
This would prevent having to kill all states as in the "pfctl -F state" command.
Phizix
-
-
-
It can take a bit for someone who actually uses the same options as you to come along.. Most here are volunteers.
Since you appear to be chugging along at diagnosis many of us will read and learn and then go off and think about it without commenting.
-
No further diagnosis to do. It is broken until they get around to fixing it.
But I am looking for a way to kill just the pertinent states without the inelegant solution of killing all states everywhere. So at least it is a more realistic workaround. As it is, if my wife is on a game or other logged in session, when any schedule expires, it kicks her (or me) off of what we were doing and we have to rejoin.
I had hoped that moving to "prosumer" equipment, that this would have worked correctly. I hope this does not offend anyone. This is as opposed to a $15K piece of equipment from one of the equipment giants.
As I mentioned, I added my comments to the bug reported on redmine in July of last year (2019). The same issue was brought up on the forums in February of 2017. I suspect that the poster may be the one who posted the bug on redmine, but I have no way of knowing, just guessing from the verbiage in both the post and bug report.
In any case, I was just having some fun with the pokes via cultural references (Simpsons and Pink Floyd).
I am actually quite disappointed, because the schedule is the primary reason I made the move to pfSense. I like a lot of things about it, but the firewall creating states to allow traffic back in via the intermediate state is very bad and a security risk.
In any case, no worries, I was just poking - LOL.
Phizix
-
It sounds like there is a way to run processes (even Python) on the box. If I had a script that could read all the states just before the schedule expires and kill all states to and from the hosts on that rule, plus any WAN rules that have them as the "original source", then I could make it work.
Phizix
-
Did you try the workaround in the thread you posted? https://forum.netgate.com/post/679884
"have a scheduled pass rule (which you already have) that expires and kills state at the same time as the scheduled block rule becomes active"
(or permanent block and scheduled allow, further down...https://forum.netgate.com/post/680667)I haven't used schedules all that much sorry, just once for limiters.
-
@teamits (Steve) ,
That is what I already have. I have a scheduled pass rule followed immediately by a block for that same alias group that is in the schedule.
The issue is that although the schedule expiring kills all states directly to and directly from the IPs in the alias, it does NOT kill the "intermediate" state where the firewall is the source and the "original source" is one of the IPs in the alias. This intermediate state appears to be spawned under one of the auto-generated NAT rules and is not subject to the state killing under the schedule expiration.
It is this intermediate state that then allows the firewall to spawn a new state from the information in that intermediate state that re-establishes the link back to the machine in the list. As far as I can tell it spawns this return state under another one of the auto-generated NAT rules, which is NOT governed by the scheduled rule at all.
Thank you for the suggestion.
Phizix
-
Actually, I don't know for sure these "hidden rules" are outbound NAT rules. I am only guessing. I can only see what the "pfctl -vvsr" command reports for rules in question as shown below:
@64(1000008011) pass out route-to (igb0 47.SPC.MY.GW) inet from 47.SPC.MY.IP to ! 47.SPC.RNG.0/21 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
@63(1000007913) pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
Phizix
-
@Phizix said in SG-5100 UDP States still alive after schedule expires. [2.4.5-RELEASE (amd64)]:
Actually, I don't know for sure these "hidden rules" are outbound NAT rules. I am only guessing. I can only see what the "pfctl -vvsr" command reports for rules in question as shown below:
@64(1000008011) pass out route-to (igb0 47.SPC.MY.GW) inet from 47.SPC.MY.IP to ! 47.SPC.RNG.0/21 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
@63(1000007913) pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
Phizix
Do you have any proxy package or something like Squid filtering running on the firewall? If so, maybe that is involved in the "hidden" rule state you mention.
-
I have three packages installed on the firewall -- pfBlockerNG-devel, Snort, and Cron. Beyond that i have no other packages or proxy installed.
pfBlockerNG-devel is blocking ads and some other lists I subscribe to.
Snort is set only to report suspicious traffic, but not block anything.
Cron is to set up the "pfctl -F state" to kill all states when the schedule expires.
I am guessing they are the auto-generated NAT outbound rules because they are part of the outbound mapping from my local IPs to the destination IPs.
You can look at the complete rules posted in the beginning rules. There are other rules I can't account for, but these are the two rules connected with the communication to internet sites under the scheduled allow rule. One rule maps from the LAN IP to the external IP, but then the firewall uses rule "64" to create the intermediate state that maps from the Firewall's external IP (from Spectrum) to one of the websites in question. In the state report under Diagnostics\States it shows the local LAN IP as the "Original source" IP, but the Firewall's external IP as the "Source" IP. The ports matches the state created under the allow by schedule rule, but of course the outbound port from the firewall has a NAT port from the firewall on that rule.
Does that clarify or muddy the waters?
Phizix
-
@Phizix said in SG-5100 UDP States still alive after schedule expires. [2.4.5-RELEASE (amd64)]:
I have three packages installed on the firewall -- pfBlockerNG-devel, Snort, and Cron. Beyond that i have no other packages or proxy installed.
pfBlockerNG-devel is blocking ads and some other lists I subscribe to.
Snort is set only to report suspicious traffic, but not block anything.
Cron is to set up the "pfctl -F state" to kill all states when the schedule expires.
Iam guessing they are the auto-generated NAT outbound rules because they are part of the outbound mapping from my local IPs to the destination IPs.
You can look at the complete rules posted in the beginning rules. There are other rules I can't account for, but these are the two rules connected with the communication to internet sites under the scheduled allow rule. One rule maps from the LAN IP to the external IP, but then the firewall uses rule "64" to create the intermediate state that maps from the Firewall's external IP (from Spectrum) to one of the websites in question. In the state report under Diagnostics\States it shows the local LAN IP as the "Original source" IP, but the Firewall's external IP as the "Source" IP. The ports matches the state created under the allow by schedule rule, but of course the outbound port from the firewall has a NAT port from the firewall on that rule.
Does that clarify or muddy the waters?
Phizix
Well, it would indicate that your original hypothesis is more likely correct. Unfortunately I don't know how you can address it short of writing your own script of some type (maybe in PHP) to do sort of semi-automatically what you did manually to connect the dots of states to LAN device IPs and then kill states associated with those connections.
This may not be a high priority issue for the developers in the grand scheme of things. While important to you I'm sure, and maybe to other users that use the scheduled rules feature, I doubt that is anywhere near a majority of pfSense users. Thus other things may be perceived as more pressing and grab the developers' time. Also don't forget this is free software. Netgate and the developers make their money from hardware and the paid support and stuff like TNSR, not from the Community Edition of pfSense. That's not to say they will never be interested, but it's likely a degree of priority thing.
If you have the skills to create the fix and post it as a pull request to Github for pfSense, I'm sure they will take a look and possibly merge it into the next version. That's kind of the idea behind open-source community-supported software like pfSense.
-
I did not expect that it would be elevated to top priority, that is why I am asking about other methods.
I also understand where they make their money. That is why I bought the SG-5100 instead of some third party hardware.
I will look at learning php for a workaround, and maybe at some point explore the code on Github. I can conceptually write up what the process that needs to spawn when a schedule ends should do, but I don't know the language enough at this point to write code.
In fact here is what it should do conceptually at the expiration of a schedule. It should first grab a snapshot of all states spawned under the rule. Then it should look for secondary related states - i.e. that match the LAN IP:Port to NAT states that have the same LAN IP:Port as the "Original source" and kill all of them too.
I really like the power of the pfSense software, but I am not a fan of rules that are not easy to connect back to a GUI rule object, even if it is not editable. For example, all of the firewall rules can be traced by looking at the Tracking ID in the rule or in the hover over and then using "pfctl -vvsr" and searching for the ID. But for the auto-generated outbound NAT rules no Tracking ID can be accessed in the GUI to trace. Also, by the way, the rules we are talking about are not in the backup xml created when you save.
I will keep plugging at it. Overall, it is still a damn sight better than any consumer firewall.
However I will again point out that regenerating a return state when the outbound state and matching inbound state that spawned the intermediate firewall state in the first place is killed just seems like very bad behavior indeed for any firewall software. To me this is the main takeaway irrespective of the schedule expiration.
In point of fact, I can recreate the same situation without the schedule expiration, by killing the state from a LAN IP:Port mapped to a Internet IP:Port, and the matching return state from the Internet IP:Port to the LAN IP:Port. The intermediate rule from the firewall itself to the Internet IP:Port will then cause a return state to be spawned back to the LAN IP:Port. This seems like a hack-able vulnerability to me.
Cheers, and have a great weekend!
Phizix
-
All,
After running for a week with the "pfctl -F state" method, it seems that most of the TV's never miss a beat since they seem to have enough buffered to keep going while reconnecting. My Amazon devices on the other hand are unresponsive for several minutes while reestablishing connection. DOH!
And my wife, that is another matter altogether. She is pretty unhappy when it boots her out of what she is doing at the time and she has to reconnect.
"C'est la vie."
Phizix
-
This is exactly what I've experienced since I recently decided to move over pfSense.
I went throughout studying, spending lot of time and money to be able to add security and properly schedule the use of internet at home, I've been able to achieve a lot, then I hit this wall.It didn't take long for people to realize they can establish a VPN connection just before the schedule kick in, opening the door to everything again...
I encourage more people to report it, with the hope dev raise the priority and finally resolve this long known issue.
Thanks for your effort, please let me know if you find a solution.pfSense 2.4.5-RELEASE (amd64) built on Tue Mar 24 15:25:50 EDT 2020 FreeBSD 11.3-STABLE
Intel i3 box, AES-NI enable, 4 Eth (1 WAN, 3 LAN)molepy
-
Thanks for the reply. What I am currently using works, but is VERY KLUGE.
I hate that it kill all states, but that is the only way to make it work.
Phizix
