Firewall has gone erratic, respecting the rule one second, stopping the next
-
The top rule and the second one are not identical. The top rule applies to traffic on the LAN interface and the second one to traffic on the WLAN interface. The rules that you posted above are for the LAN interface and it would appear that the top rule worked correctly. However, we can't tell what happened for the second rule without knowing what the rest of the rules are for the WLAN. Without seeing the rest of the WLAN rules in context to the one that blocked the traffic it's impossible to know if the blocking was correct or not.
What I find puzzling is why there are packets on two different interfaces, using the same ip address and ports, from the same client. What is the WLAN and what is it's IPV4 address in the Interface Settings?
Something strange is definitely going on here. The top rule entries that end in 617 is your LAN:pass VPN_WAN log rule on the LAN interface. As far as I can tell that rule should have routed the traffic out the interface and therefor nothing would have been blocked by the 943 rule, which is also on the LAN interface but below the 617 rule. It's strange that the 617 rule also does not display the description at times. I'm wondering if it possible that the 617 rule is partially corrupted. It might be worth deleting the rule and recreating it from scratch. -
@cool1two Can you share a screenshot of the logs with the rule description/number displayed. You can do that by clicking the little wrench icon in the upper right when you have the log displayed and enabling this setting:
This way we can confirm whether or not it's the same rule that's causing the issue. -
@dma_pf
Sure, I can post the current output. However, that setting only works for the normal view and not dynamic.I can tell you that today, after the reboot yesterday, the same rule is working for all of my interfaces. Yesterday, when the IOT network was being blocked, it was the default WAN rule (Default deny rule IPv4 (1000000103)) that was being hit.
-
@dma_pf said in Firewall has gone erratic, respecting the rule one second, stopping the next:
The top rule and the second one are not identical. The top rule applies to traffic on the LAN interface and the second one to traffic on the WLAN interface. The rules that you posted above are for the LAN interface and it would appear that the top rule worked correctly. However, we can't tell what happened for the second rule without knowing what the rest of the rules are for the WLAN. Without seeing the rest of the WLAN rules in context to the one that blocked the traffic it's impossible to know if the blocking was correct or not.
What I find puzzling is why there are packets on two different interfaces, using the same ip address and ports, from the same client. What is the WLAN and what is it's IPV4 address in the Interface Settings?
LAN is a bridge, that joins ELAN (Ethernet LAN) and WLAN (Wifi LAN). I want the wired and wireless LAN to share addresses and DHCP server. I saw several tutorials that had this set up. Is this a problem?
Something strange is definitely going on here. The top rule entries that end in 617 is your LAN:pass VPN_WAN log rule on the LAN interface. As far as I can tell that rule should have routed the traffic out the interface and therefor nothing would have been blocked by the 943 rule, which is also on the LAN interface but below the 617 rule. It's strange that the 617 rule also does not display the description at times. I'm wondering if it possible that the 617 rule is partially corrupted. It might be worth deleting the rule and recreating it from scratch.I did, still nothing.
I had not suffer from this issue until today again. What I realized is that I could not save the LAN interface again. It gave me a warning about RA. The funny part is that I have IPv6 disabled, so not sure how RA got activated in that interface. So I gave LAN a random IPv6 address, which allowed me to access the RA configuration of LAN. I disabled it, removed the random IPv6 address, and then suddenly, the firewall was working as expected again. I had never configured or even gone into the DCHPv6 configuration. I am not sure if that misconfiguration was confusing the firewall or if removing that trigger something in the firewall to compile the rules again properly.
If the problem comes back I will report back.
-
@cool1two said in Firewall has gone erratic, respecting the rule one second, stopping the next:
I can tell you that today, after the reboot yesterday, the same rule is working for all of my interfaces. Yesterday, when the IOT network was being blocked, it was the default WAN rule (Default deny rule IPv4 (1000000103)) that was being hit.
That would imply that a device outside of pfSense, with an IP address of 192.168.55.1 (non-routable RC1918 address), was trying to enter through the WAN. The default deny WAN rule would block all traffic trying to enter through the WAN interface which was not a response to traffic originating from inside of pfsense. How do you connect to your ISP? Do you have a router in front of pfsense? Where is the 192.168.55.xxx network being assigned?
-
I would almost agree with you if the connections had continued to be blocked after the reboot. If it were some external device on the WAN side of the PFSense, then I would not expect my reboot to affect its connection attempts. The PFSense is the edge of my network. The ISP modem does not use NAT, Wireless, or any other function aside from handing the PFSense a routable WAN IP.
The 192.168.55.xxx network is the IP range for the IOT VLAN. PFSense handles DHCP and routing for the range, and 192.168.55.1 is "gateway.iot" an IP address on the PFSense itself.
I think it is more likely that the firewall rules were reloaded when I wasn't looking, possibly by the watchdog or an interface flapping. After the reload, the rules were simply out of order or missing. I should have checked for this at the time the issue was occurring. Perhaps next time.
-
It has happened again for a few hours today. The rules are not being respected, the pass rule with the gateway is ignored and the last block all rule gets triggered, hence all Internet is blocked. I left pfSense alone for a few hours and now it is working again. Not touched a thing, just went to watch a movie. Honestly, I can not be dealing with the firewall misbehaving like this.
The firewall was even triggering a rule that I deleted, even after rebooting pfSense.
I have installed pfSense from zero several times, and the same thing keeps happening. Either I am doing something wrong which I do not understand or I am triggering a bug in pfSense. The thing is I can not continue like this, I need to find what is wrong or stop using pfSense.
-
Please take a look at your System Logs under Status. As I mentioned in my previous post, I suspect that a reload of the firewall rules could align with your issues. The logs would look something like below. I have not had a repeat of the problem. However, I now know what firewall filters I need to use to identify if the issue is occurring or not.
-
@cool1two You are right, there is something there. This is the log. The VPN_WAN_GW, the one where the traffic is supposed to go out through the VPN, seems to go down regularly every few minutes. What you see in the log I am posting gets repeated every few minutes. Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available?
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
This is the log:
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available
That was a complaint in IPv6 No Gateway after 2.5 upgrade and the Redmine report but that's specifically related to IPv6.
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
I have installed pfSense from zero several times, and the same thing keeps happening. Either I am doing something wrong which I do not understand or I am triggering a bug in pfSense. The thing is I can not continue like this, I need to find what is wrong or stop using pfSense.
I'm confident that the issues you've been having getting your network setup are not related to a bug in pfsense. In the other thread that I am helping you on you mentioned that you are new to pfsense. Pfsense is very robust and flexible and therefore can be complex to setup. It's very easy to overthink and over complicate it's setup. Just because it has a lot of "bells and whistles" doesn't mean you have to necessarily use them. The more more complex the setup the more room for configuration errors.
Given the issues you've highlighted in both of your postings, it would be very helpful if you could post a drawing of your network. It doesn't have to be pretty, just draw it out by hand and post a picture. As to the router, please indicate the make and model and any additional function it provides via the ISP (bundled services like tv, phone, wireless). Also make sure you indicate all wireless access points and switches.
Can you also post screen shots of the firewall rules for ELAN and WLAN?
-
It is indeed a problem with the gateway going down. After the gateway goes down, the firewall ignores the rules that have the gateway that went down. Hence why it seemed to randomly start ignoring some rules.
Here is the details with the gateway logs at the end. It seems that the problem is that I am getting high latency and high packet loss from the VPN connection. I guess I will have to go and ask in the VPN section. Thanks everybody for the help.
The interface of this gateway is up:
And the VPN connection seems fine:
But to get it to work again I have to restart the VPN connection. Here is what it is happening:
And then after some time:
I will open a doubt in the VPN section. But ideally someone would indicate me how can I losen the gateway so it accepts high packet loss until I can fix the issue with the vpn. I would prefer to have a bad connection than not connection at all.
-
@teamits said in Firewall has gone erratic, respecting the rule one second, stopping the next:
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Is it possible that the firewall rule using the VPN_WAN_GW gets ignored because the gateway is not available
That was a complaint in IPv6 No Gateway after 2.5 upgrade and the Redmine report but that's specifically related to IPv6.
Well, I can tell you it is happening in 2.5 with IPv4, that is for sure.
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
You can test this by disabling the Gateway Monitor in System/Routing/Gateways for the VPN Gateway. If the IP address that the monitor is using does not respond to the monitoring service then the gateway will be marked as down even if the connection is stable.
-
@dma_pf said in Firewall has gone erratic, respecting the rule one second, stopping the next:
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
How do I go about debugging what might be happening with this gateway? The logs of OpenVPN, the vpn client I am using, show nothing like that, only show the initial connection and nothing more, and the VPN status says it has been connected without issues. I would appreciate any advice on how to look for the issue.
You can test this by disabling the Gateway Monitor in System/Routing/Gateways for the VPN Gateway. If the IP address that the monitor is using does not respond to the monitoring service then the gateway will be marked as down even if the connection is stable.
Ok, it seems that fixed the issue of the Gateway going down all the time. I guess now I should go to the VPN section and find out why I have 20%+ packet loss, because I am guessing that is not normal. I am hoping it is a problem with the monitoring and not with the VPN itself.
Again, thanks everybody for helping someone who is starting. I never though this would turned to be so involved.
-
@dma_pf
Here to help the cause. Below is a network diagram and screenshots of my rules. There is nothing here that is exotic. Random comments and suggestions welcome :)Network:
FW Floating:
FW WAN:
FW LAN:
FW IOT:
FW NGT - Test Network:
FW NAT:
-
@cool1two said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Random comments and suggestions welcome
You rules allowing dns to XYZ "net" don't make a lot of sense. You should use XYZ "address" here.. For example the iot network... Pfsense has no control over A talking to B in the iot net.. So saying your allowing to iot net doesn't make sense, what you can allow is dns to pfsense iot address..
edit: Also not really understanding the point of the multicast and and broadcast traffic rules on your floating.. Other than not logging it.. What for example is pfsense going to do with LLMNR multicast traffic? Or the SSDP stuff? Other than not wanting that logged in the default block rule - Not sure what your expecting those rules to do?
-
@dicmo said in Firewall has gone erratic, respecting the rule one second, stopping the next:
Ok, it seems that fixed the issue of the Gateway going down all the time. I guess now I should go to the VPN section and find out why I have 20%+ packet loss, because I am guessing that is not normal. I am hoping it is a problem with the monitoring and not with the VPN itself.
I had a similar situation with a VPN provider. I got in touch with them and they suggested using the WAN IP address of their server as the monitor IP. That didn't work. They suggested using the IP address of their DNS servers. That didn't work either. Nonetheless running a continuous ping test through their VPN servers out to the internet to various servers all showed acceptable latency and packet loss. It was clearly an issue in ping responses times to their servers which was causing the issue, but it had nothing to do with the quality of their vpn services.
My "fix" was to create 2 other simultaneous connections to servers offered by my VPN provider. (Just duplicate the interfaces, gateways, NAT, firewall, etc. of the existing VPN but point them to different servers.). I disabled the gateway monitoring on each of the gateways and then bound the 3 interfaces as an interface group. You can then use the interface group for policy routing in your firewall rules. Since then, I've never seen a gateway marked as down, and if one of the VPN servers was to go down then traffic would still be able to get routed out the other 2 servers. I've never been in a position to where I've not been able to route traffic through the VPN provider.
-
@johnpoz
Rules 1 and 2 in the VLAN in effect say, if you are using the VLAN gateway as your DNS provider then you can pass, otherwise you will be blocked.This prevents IOT systems with pre-programmed DNS servers, such as Google Mini, from using an unauthorized DNS server.
As far as the IGMP floating rules. All of the switches participate in IGMP Snooping Query election for the network. Additionally, all of the switches are doing LLDP discovery. The Netgate PFsense router was blocking this traffic for all VLANs and for any device on those VLANs. This caused several issues, including all of the switches believing they had been elected as the IGMP Snooping Querier. With these rules in place all of the switches, including the UNIFI now agree on who is doing what.
.25 switch
.26 switch
.27 switch
-
@johnpoz
I did take your advice and switch the VLAN rules to IOT and NGT address instead of Net. Thank you for the suggestion.
