Gateway/firewall falling over when a nessus scan has been run on the network
-
Since nessus sends a lot of individual connections to hosts its scanning it sounds like you might be running out of states in pf. This would prevent new connections from going to/through the pfsense box even though the box itself is up. I'd try increasing the maximum number of states available provided you have the RAM for it (each state needs about 256 bytes IIRC) or change the state optimization mode to aggressive.
-
Hi,
Thanks for your reply, I have actually done both of those things already. it did help and the states are being cleared quick enough (plenty of space for new ones) but it still drops it just takes longer for the firewall to drop.
It can't handle the amount of connections going through it, I'm not sure what else I can tweak to fix this, we don't have problems through other firewalls?
Thanks for your help
-
Some of the largest PCI ASV scanners in the world use pfSense at very large scanning scale, it's not a general problem. It definitely sounds like state exhaustion, which can happen in multiple places. Sounds like it's not on your firewall if you definitely have your states high enough, but could be something upstream from it like your modem.
-
The other thing you could do is bypass state tracking entirely for your Nessus device, although you will need to pretty much allow everything to/from that device in pfSense (not sure how applicable this would be to your security policy).
You could create 2 floating rules, one allowing everything FROM your Nessus device and another allowing everything TO your Nessus device. Set them to quick and in the advanced options set state tracking to none. This will effectively bypass the state table for all traffic to/from the Nessus scanner, but again be aware of the security implications of this as pfsense would filter absolutely no traffic for your Nessus scanner.
-
Hi,
Thanks for your replies, any idea where in advanced you can disable the state tracking, Ideally I would rather not create an any any rule, but for a quick test it might be ok. but even so will this disable state tracking?
When I login back in to the pfsense from the mgmnt port after the network has crashed the states are about 100/1900000 so I don't think it is to do with the states, I doubled the memory to 4gb and added a second processor the hardware isn't the issue either.
I am running out of ideas here.. :(
Thanks for your help
-
I just set state type to none in the firewall but after that no connection could be made to the internet so I had to put it back to keep state.
Any other suggestions??
Thanks
-
I can't tell 100% from your replies but it sounds like you created a single any-any rule with state tracking disabled. If this is the case, this is not what you want to do.
You'll want to create 2 floating rules. One with a source of your Nessus scanner IP and destination of any, TCP/UDP protocols, enable quick, state tracking set to none. Second rule would be a source of any and destination of your Nessus scanner IP, same setup as the above rule.
Sorry if this is actually what you did, from your post its difficult to tell. You might want to post a screenshot of the rules you are creating so they can be looked at.
-
It's impossible to do no state in most circumstances like that (anything with NAT), don't do it.
-
Any other suggestions??
I haven't seen a follow up to: @deej1:
- I will look into these logs more and post again, the logs on the pfsense webgui had nothing in them.
Note though that you need to look at the logs BEFORE rebooting pfSense because some versions of pfSense clear the logs on reboot.
Please also post the output of pfSense shell commands:```
/etc/rc.banner
ifconfig -a
netstat -m
What version of pfSense are you using (from report on the home page)? You have reported that when the "fall over" condition occurs you can access the pfSense console, on the pfSense console get ping responses from internet hosts, have plenty of free states, client computers can't access the internet. This suggests to me a possible connectivity problem between client computers and pfSense. That this connectivity problem is apparently rectified by rebooting pfSense suggests it is probably in pfSense rather than the client(s?) or something between client and pfSense. What is between the clients and pfSense? Does the problem affect all clients? When pfSense has "fallen over" 1\. What is reported on the client when you attempt to ping an internet host by IP address (say 8.8.8.8, a Google name server)? 2\. What is reported on the pfSense console when you attempt to ping a client computer? -
Hi,
Sorry the logs are always empty, they don't have anything useful in them at all.
Version:
2.0.1-RELEASE (i386)
built on Mon Dec 12 17:53:52 EST 2011
FreeBSD 8.1-RELEASE-p6$ ifconfig -a bge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:16:35:9f:db:10 inet 10.66.3.1 netmask 0xffffff00 broadcast 10.66.3.255 inet6 fe80::216:35ff:fe9f:db10%bge0 prefixlen 64 scopeid 0x1 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active bge1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:16:35:9f:db:11 inet6 fe80::216:35ff:fe9f:db11%bge1 prefixlen 64 scopeid 0x2 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9 <rxcsum,vlan_mtu>ether 00:08:02:cd:57:94 inet 10.66.0.3 netmask 0xfffffff8 broadcast 10.66.0.7 inet6 fe80::208:2ff:fecd:5794%fxp0 prefixlen 64 scopeid 0x3 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>) status: active fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9 <rxcsum,vlan_mtu>ether 00:08:02:cd:57:95 inet 10.66.0.9 netmask 0xfffffff8 broadcast 10.66.0.15 inet6 fe80::208:2ff:fecd:5795%fxp1 prefixlen 64 scopeid 0x4 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>) status: active lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460 syncpeer: 224.0.0.240 maxupd: 128 syncok: 1 pflog0: flags=100 <promisc>metric 0 mtu 33200 enc0: flags=0<> metric 0 mtu 1536 bge1_vlan1101: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 00:16:35:9f:db:11 inet6 fe80::216:35ff:fe9f:db10%bge1_vlan1101 prefixlen 64 scopeid 0x9 inet 10.66.1.1 netmask 0xffffff00 broadcast 10.66.1.255 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1101 parent interface: bge1 bge1_vlan1105: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 00:16:35:9f:db:11 inet6 fe80::216:35ff:fe9f:db10%bge1_vlan1105 prefixlen 64 scopeid 0xa inet 10.66.5.1 netmask 0xffffffc0 broadcast 10.66.5.63 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1105 parent interface: bge1</full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast>$ netstat -m 1436/1894/3330 mbufs in use (current/cache/total) 1435/1595/3030/25600 mbuf clusters in use (current/cache/total/max) 1434/998 mbuf+clusters out of packet secondary zone in use (current/cache) 0/90/90/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 3230K/4023K/7253K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/8/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routinesThere is switch between the pfsense and users.
The problem affects all clients.
The client computers lose there IP and any network connectivity.
The pfsense box cannot ping 8.8.8.8,
Hope that is enough info, Thanks for your help
-
Sorry the logs are always empty, they don't have anything useful in them at all.
I find this so surprising I need clarification: When you write "the logs are empty" do you mean "the logs contain nothing at all" or do you mean "the logs don't report anything that seems relevant to this particular problem"?
Hope that is enough info, Thanks for your help
Thanks for the additional information. Unfortunately it is not enough for me to be able to identify the problem.
When I asked @wallabybob:
When pfSense has "fallen over"
1. What is reported on the client when you attempt to ping an internet host by IP address (say 8.8.8.8, a Google name server)?
2. What is reported on the pfSense console when you attempt to ping a client computer?I was looking for more details than @deej1:
The pfsense box cannot ping 8.8.8.8,
Ping can report a number of different errors and the exact text of the report contains considerably more information than the high level summary "cannot ping". Please provide the details I asked for.
Your report @deej1:
The pfsense box cannot ping 8.8.8.8,
seems to contradict your earlier report that you can ping from pfsense shell to an external website. Maybe the details of the ping response will explain the apparent contradiction. Can you explain this apparent contradiction? (Note I get ping response from 8.8.8.8 over the public internet.)
None of your pfSense interfaces has a public IP address. So what is between pfSense and the targets of the nessus scan? What is between pfSense and the public internet?