Recurring Default deny rule IPv4(1000000103)
-
I know this has been covered and I have been reading posts the last 2 hours trying to nail down why this is happening and how to fix/prevent going forward. I have been running on Pfsense since back in the 2.0.1 days and haven't come across this problem before. I switched this location to a new router running 2.7.2. They had been running an old version 2.1.5
, but now at least fully updated. I copied over the configuration and have been wrestling with the SIP settings the past few days but finally got everything working yesterday morning. Using NAT for port forwarding to several servers behind pfsense with a static ip on the WAN side. I still have the old router up, so I am able to swap back if something breaks.So I got everything up and ran great throughout yesterday and first part of the morning then at 13:00 on the pfsense clock it started showing this "Default deny rule IPv4(1000000103) and no machines on the network had internet access. I was not able to ping google DNS at 8.8.8.8. I logged into pfsense and tried pinging straight from the router and it would not ping. I swapped the network back to the old router, everything worked, swapped back to new router, no internet. Left it on the old router for about 30 mins, went back to the new router and everything is working properly again on it.
From reading around I am assuming it is Out of State packets, but I never really found anything explaining in simple terms why it happened and how to avoid it.
I hope all of this makes sense. Thank you.
-
@rwarnken and everyone of those blocks is out of state.. ie they are not syn.. they are ack and fin,ack... So yeah they are going to be denied.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/log-filter-blocked.html
-
@johnpoz So I looked at that link before. My assumption is going to the allow all rule for LAN and changing the TCP Flags? Should I go with "Any Flags" or only use a specific flag?
The link helped me ID it initially but I was not in-tune enough with the firewall settings to know what to change to prevent it from happening.
Greatly appreciate the input!
-
@rwarnken no that is not the solution.. The solution to your problem is why is client not sending syn to pfsense when it wants to talk to one of those devices.. You got something messed up..
Did you just flush your states or something, and that is why you are seeing those?
Since I don't see any SA in there, it doesn't say to me asymmetrical traffic.. What it looks like to me is states were flushed.. And your seeing the client sending hey here is my ack, why do you not answer - ok let me close this connection with FA and I will start a new conversation with syn..
You could see this for example if your gateway on pfsense went down and you have it flush the states when that triggers, etc.
-
@johnpoz Would swapping from one router to the other possibly cause this? For testing I just unplugged the WAN and LAN cables from the old router and plugged them into the new router. I haven't turned either router off or rebooted either of them. Both routers are set to 192.168.1.1.
It randomly happened this morning, I hadn't even made it to my desk yet this morning and it just went out on everyone. No issues yesterday and ran for the entire day. Initially I had flushed states etc. when troubleshooting SIP several days earlier.
When you mention flushing states if the gateway(i assume the WAN gateway) goes down, where would I check for this?
Would the possible mess up be in the firewall rules?
I thought I was fairly savvy with pfsense but apparently it is telling me I have more to learn. Thank you!
Added:
@rwarnken Looking at the rules and mentioning states, I see next to the rule protocol it show 2.903k/2.64GiB under states. Does this seem normal? I didn't think to check this when it wasn't working so not sure if it triggered a flush. -
@rwarnken About 2 hour later and Firewall/Rules/LAN showing 2.589K/11.10GiB under States. Seems like a big jump in size for 2 hours of operating.
-
@rwarnken well that left is how many active states you have open, and the right is how much data has gone through the rule..
How many clients do you have? How many states is that, is that 2600?
So you moved from one to the other - and it has the same IP? Then yeah clients not knowing their gateway changed would just continue sending data, but pfsense never saw the syn to open a state, so yeah it would block that traffic.
-
@johnpoz Probably around 80 total clients.
State table size is showing 2320/390000
Yes from one to the other, same ip on both. The new router used the backup file from the original router.
-
@rwarnken well that would explain it then - those blocks should of died off by now.. Saw quite a few fin,acks Are you still seeing them?
Such blocks are somewhat normal - see it when for example a stupid phone thinks hey I just moved from cell data I can continue with the same conversation over wifi, or some device has been in standby for 6 hours - and thinks hey this session should still be open ;)
once client doesn't get an answer, it will just open a new session with a syn which pfsense will open a state and then allow traffic.. That is if the traffic is allowed.. If you see a bunch of syn,acks blocked by the default rule - this points to asymmetrical traffic flow..
-
@johnpoz Things look clear still this morning with normal operations. I am not seeing and fin, acks. All the bocks in the firewall are coming from the WAN, which to me is normal/typical. Again, I am open to any suggestions, but for right now I think just keep an eye on it.
-
@rwarnken Like I said for what you did, seeing a bunch of those out of state blocks should be expected.. And even now and then seeing some would be within normal operation..
So seeing a few of those now and then can be expected, especially on a larger network with lots of different clients.. What I would be concerned with is seeing SA.. Because seeing those point to some sort of asymmetrical traffic flow..
-
and remember, as soon as you have sorted out things, go for the obvious :
Un-check :
and appreciate the silence.
After all, what you can't see doesn't exist ^^ -
So far things seem to be going normal. The help is greatly appreciated.
-
@rwarnken as @Gertjan mentions, turning off logging of the default deny can be helpful for keeping your logs less busy.
I have it off, and just have the stuff I am interested in logging per settings on the rules, etc.
If you run into something not working and you need to troubleshoot to see if say its being blocked by default deny, turning it back on is just a click away.
