Firewall has gone erratic, respecting the rule one second, stopping the next
-
I am new to pfSense and I do not know how to make sense of what pfSense is doing. I have had pfSense running as a VM on a linux host for some days now. It receives the two NICS as virtual NICS from linux. Today in the last hours it has started behaving erratically, blocking and passing packets for no good reason without the rules changing. I have not changed any configuration in the pfSense since yesterday.
This is the logs of the firewall. Notice how packets with the same origin and destination (same ports, same protocols) pass and gets blocked, sometimes within the same second. During the period these logs were generated the firewall rules did not change.
Also at some point the log forgets the description of a rule and leaves only the numberic ID. When I reload the logs the description is back there. Without touching the rules at all. What could be going on?
First the rules:
Here some examples in the logs of what I am talking:
Let me reiterate again that during all this period the rules have not changed. Exactly the same. Also, tried rebooting pfSense, still same behaviour.
Any idea?
-
I left pfSense alone since I had no idea why that was happening. No reboot, no change of rules, no nothin, I just left it alone. After an hour, pfSense is working fine again.
Nobody else is experiencing something similar?
-
I must say I am having a very similar issue. Today one of my VLANs is being blocked as WAN. At the same time, IPs in the VLAN are being passed by the same rule that is being blocked.
Rules:
VLANS:
Firewall logs:
While the source port is different, the way the firewall rule is setup it should make no difference. You can see the blocked and allowed are in the same /24 subnet.
This was literally working yesterday, and not working today. No Reboots or changes. Any input would be appreciated.
-
This was causing some major issues for me so I took the drastic step of restarting the firewall. After the restart everything seems back to normal.
-
The top rule and the second one are not identical. The top rule applies to traffic on the LAN interface and the second one to traffic on the WLAN interface. The rules that you posted above are for the LAN interface and it would appear that the top rule worked correctly. However, we can't tell what happened for the second rule without knowing what the rest of the rules are for the WLAN. Without seeing the rest of the WLAN rules in context to the one that blocked the traffic it's impossible to know if the blocking was correct or not.
What I find puzzling is why there are packets on two different interfaces, using the same ip address and ports, from the same client. What is the WLAN and what is it's IPV4 address in the Interface Settings?
Something strange is definitely going on here. The top rule entries that end in 617 is your LAN:pass VPN_WAN log rule on the LAN interface. As far as I can tell that rule should have routed the traffic out the interface and therefor nothing would have been blocked by the 943 rule, which is also on the LAN interface but below the 617 rule. It's strange that the 617 rule also does not display the description at times. I'm wondering if it possible that the 617 rule is partially corrupted. It might be worth deleting the rule and recreating it from scratch. -
@cool1two Can you share a screenshot of the logs with the rule description/number displayed. You can do that by clicking the little wrench icon in the upper right when you have the log displayed and enabling this setting:
This way we can confirm whether or not it's the same rule that's causing the issue. -
@dma_pf
Sure, I can post the current output. However, that setting only works for the normal view and not dynamic.I can tell you that today, after the reboot yesterday, the same rule is working for all of my interfaces. Yesterday, when the IOT network was being blocked, it was the default WAN rule (Default deny rule IPv4 (1000000103)) that was being hit.
-
@dma_pf said in Firewall has gone erratic, respecting the rule one second, stopping the next:
The top rule and the second one are not identical. The top rule applies to traffic on the LAN interface and the second one to traffic on the WLAN interface. The rules that you posted above are for the LAN interface and it would appear that the top rule worked correctly. However, we can't tell what happened for the second rule without knowing what the rest of the rules are for the WLAN. Without seeing the rest of the WLAN rules in context to the one that blocked the traffic it's impossible to know if the blocking was correct or not.
What I find puzzling is why there are packets on two different interfaces, using the same ip address and ports, from the same client. What is the WLAN and what is it's IPV4 address in the Interface Settings?
LAN is a bridge, that joins ELAN (Ethernet LAN) and WLAN (Wifi LAN). I want the wired and wireless LAN to share addresses and DHCP server. I saw several tutorials that had this set up. Is this a problem?
Something strange is definitely going on here. The top rule entries that end in 617 is your LAN:pass VPN_WAN log rule on the LAN interface. As far as I can tell that rule should have routed the traffic out the interface and therefor nothing would have been blocked by the 943 rule, which is also on the LAN interface but below the 617 rule. It's strange that the 617 rule also does not display the description at times. I'm wondering if it possible that the 617 rule is partially corrupted. It might be worth deleting the rule and recreating it from scratch.I did, still nothing.
I had not suffer from this issue until today again. What I realized is that I could not save the LAN interface again. It gave me a warning about RA. The funny part is that I have IPv6 disabled, so not sure how RA got activated in that interface. So I gave LAN a random IPv6 address, which allowed me to access the RA configuration of LAN. I disabled it, removed the random IPv6 address, and then suddenly, the firewall was working as expected again. I had never configured or even gone into the DCHPv6 configuration. I am not sure if that misconfiguration was confusing the firewall or if removing that trigger something in the firewall to compile the rules again properly.
If the problem comes back I will report back.
-
@cool1two said in Firewall has gone erratic, respecting the rule one second, stopping the next:
I can tell you that today, after the reboot yesterday, the same rule is working for all of my interfaces. Yesterday, when the IOT network was being blocked, it was the default WAN rule (Default deny rule IPv4 (1000000103)) that was being hit.
That would imply that a device outside of pfSense, with an IP address of 192.168.55.1 (non-routable RC1918 address), was trying to enter through the WAN. The default deny WAN rule would block all traffic trying to enter through the WAN interface which was not a response to traffic originating from inside of pfsense. How do you connect to your ISP? Do you have a router in front of pfsense? Where is the 192.168.55.xxx network being assigned?
-
I would almost agree with you if the connections had continued to be blocked after the reboot. If it were some external device on the WAN side of the PFSense, then I would not expect my reboot to affect its connection attempts. The PFSense is the edge of my network. The ISP modem does not use NAT, Wireless, or any other function aside from handing the PFSense a routable WAN IP.
The 192.168.55.xxx network is the IP range for the IOT VLAN. PFSense handles DHCP and routing for the range, and 192.168.55.1 is "gateway.iot" an IP address on the PFSense itself.
I think it is more likely that the firewall rules were reloaded when I wasn't looking, possibly by the watchdog or an interface flapping. After the reload, the rules were simply out of order or missing. I should have checked for this at the time the issue was occurring. Perhaps next time.
-
It has happened again for a few hours today. The rules are not being respected, the pass rule with the gateway is ignored and the last block all rule gets triggered, hence all Internet is blocked. I left pfSense alone for a few hours and now it is working again. Not touched a thing, just went to watch a movie. Honestly, I can not be dealing with the firewall misbehaving like this.
The firewall was even triggering a rule that I deleted, even after rebooting pfSense.
I have installed pfSense from zero several times, and the same thing keeps happening. Either I am doing something wrong which I do not understand or I am triggering a bug in pfSense. The thing is I can not continue like this, I need to find what is wrong or stop using pfSense.
-
Please take a look at your System Logs under Status. As I mentioned in my previous post, I suspect that a reload of the firewall rules could align with your issues. The logs would look something like below. I have not had a repeat of the problem. However, I now know what firewall filters I need to use to identify if the issue is occurring or not.
-
@cool1two You are right, there is something there. This is the log. The VPN_WAN_GW, the one where the traffic is supposed to go out through the VPN, seems to go down regularly every few minutes. What you see in the log I am posting gets repeated every few minutes. Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available?
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
This is the log:
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available
That was a complaint in IPv6 No Gateway after 2.5 upgrade and the Redmine report but that's specifically related to IPv6.
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
I have installed pfSense from zero several times, and the same thing keeps happening. Either I am doing something wrong which I do not understand or I am triggering a bug in pfSense. The thing is I can not continue like this, I need to find what is wrong or stop using pfSense.
I'm confident that the issues you've been having getting your network setup are not related to a bug in pfsense. In the other thread that I am helping you on you mentioned that you are new to pfsense. Pfsense is very robust and flexible and therefore can be complex to setup. It's very easy to overthink and over complicate it's setup. Just because it has a lot of "bells and whistles" doesn't mean you have to necessarily use them. The more more complex the setup the more room for configuration errors.
Given the issues you've highlighted in both of your postings, it would be very helpful if you could post a drawing of your network. It doesn't have to be pretty, just draw it out by hand and post a picture. As to the router, please indicate the make and model and any additional function it provides via the ISP (bundled services like tv, phone, wireless). Also make sure you indicate all wireless access points and switches.
Can you also post screen shots of the firewall rules for ELAN and WLAN?
-
It is indeed a problem with the gateway going down. After the gateway goes down, the firewall ignores the rules that have the gateway that went down. Hence why it seemed to randomly start ignoring some rules.
Here is the details with the gateway logs at the end. It seems that the problem is that I am getting high latency and high packet loss from the VPN connection. I guess I will have to go and ask in the VPN section. Thanks everybody for the help.
The interface of this gateway is up:
And the VPN connection seems fine:
But to get it to work again I have to restart the VPN connection. Here is what it is happening:
And then after some time:
I will open a doubt in the VPN section. But ideally someone would indicate me how can I losen the gateway so it accepts high packet loss until I can fix the issue with the vpn. I would prefer to have a bad connection than not connection at all.
-
@teamits said in Firewall has gone erratic, respecting the rule one second, stopping the next:
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available
That was a complaint in IPv6 No Gateway after 2.5 upgrade and the Redmine report but that's specifically related to IPv6.
Well, I can tell you it is happening in 2.5 with IPv4, that is for sure.
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
You can test this by disabling the Gateway Monitor in System/Routing/Gateways for the VPN Gateway. If the IP address that the monitor is using does not respond to the monitoring service then the gateway will be marked as down even if the connection is stable.
-
@dma_pf said in Firewall has gone erratic, respecting the rule one second, stopping the next:
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
You can test this by disabling the Gateway Monitor in System/Routing/Gateways for the VPN Gateway. If the IP address that the monitor is using does not respond to the monitoring service then the gateway will be marked as down even if the connection is stable.
Ok, it seems that fixed the issue of the Gateway going down all the time. I guess now I should go to the VPN section and find out why I have 20%+ packet loss, because I am guessing that is not normal. I am hoping it is a problem with the monitoring and not with the VPN itself.
Again, thanks everybody for helping someone who is starting. I never though this would turned to be so involved.
-
@dma_pf
Here to help the cause. Below is a network diagram and screenshots of my rules. There is nothing here that is exotic. Random comments and suggestions welcome :)Network:
FW Floating:
FW WAN:
FW LAN:
FW IOT:
FW NGT - Test Network:
FW NAT:
