Gateway/firewall falling over when a nessus scan has been run on the network

deej1

Hi,

I am currently running version 2.0.1-RELEASE (i386), any time a nessus scan is run internal to an external address the pfsense seems to fall over and I cannot see the network from any computers behind the pfsense. I can access the pfsense box with a monitor and everything seems to work, I can ping addresses and see what I should be able to see but the client machines cannot connect to the box any more untill I reboot pfsense.

Any ideas what could be causing the problem or is this a bug which needs fixing? I have upped the states to 10x what it was, the states don't seem to be the problem.

Thanks for your help

wallabybob

@deej1:

Any ideas what could be causing the problem or is this a bug which needs fixing?

I wouldn't like to speculate without more information.

@deej1:

any time a nessus scan is run internal to an external address

What is a nessus scan? Do you mean the scan accesses computers on "the other side" of pFsense from the computer running the scan?

@deej1:

the pfsense seems to fall over and I cannot see the network from any computers behind the pfsense.

cannot see which network? The Internet? How are you attempting to "see" the network (ping? through a web browser?) and what is reported on these attempts?

@deej1:

I can access the pfsense box with a monitor and everything seems to work,

Do you mean you can connect a "screen" and access a shell session?

@deej1:

I can ping addresses and see what I should be able to see

ping from where to where?

@deej1:

but the client machines cannot connect to the box any more untill I reboot pfsense.

Do you mean the client machines don't seem to connect to the pfSense web server? What is reported on the connection attempt?

If you can get a shell session on pfSense you could look in the log files in /var/log/. These are "circular logs" and should be displayed by a command such as```
clog /var/log/system.log | more


Log files that might have something of interest include system.log, lighttpd.log and lighttpd.error.log
Events that might be of interest include web server exiting, and interface related events such as Link Down and timeouts. You will probably have to examine the log files after creating the condition and before rebooting pfSense. It would also be useful to post the output of the pfSense shell command```
/etc/rc.banner ; echo " "; ifconfig -a
```to report the state of the interfaces.

deej1

@wallabybob:

@deej1:

Any ideas what could be causing the problem or is this a bug which needs fixing?

I wouldn't like to speculate without more information. - No problem

@deej1:

any time a nessus scan is run internal to an external address

What is a nessus scan? Do you mean the scan accesses computers on "the other side" of pFsense from the computer running the scan? - Yes this is a scan which checks for open services or ports on computers externally.

@deej1:

the pfsense seems to fall over and I cannot see the network from any computers behind the pfsense.

cannot see which network? The Internet? How are you attempting to "see" the network (ping? through a web browser?) and what is reported on these attempts? - When it falls over any client computer on the network lost connection, no internet, no network shares etc.. (using pfsense as dhco)

@deej1:

I can access the pfsense box with a monitor and everything seems to work,

Do you mean you can connect a "screen" and access a shell session? - Yes

@deej1:

I can ping addresses and see what I should be able to see

ping from where to where? - pfsense shell to an external website

@deej1:

but the client machines cannot connect to the box any more untill I reboot pfsense.

Do you mean the client machines don't seem to connect to the pfSense web server? What is reported on the connection attempt?

If you can get a shell session on pfSense you could look in the log files in /var/log/. These are "circular logs" and should be displayed by a command such as```
clog /var/log/system.log | more
Log files that might have something of interest include system.log, lighttpd.log and lighttpd.error.log
Events that might be of interest include web server exiting, and interface related events such as Link Down and timeouts. You will probably have to examine the log files after creating the condition and before rebooting pfSense. It would also be useful to post the output of the pfSense shell command```
/etc/rc.banner ; echo " "; ifconfig -a
```to report the state of the interfaces.

**- I will look into these logs more and post again, the logs on the pfsense webgui had nothing in them.**

echoranger

Since nessus sends a lot of individual connections to hosts its scanning it sounds like you might be running out of states in pf. This would prevent new connections from going to/through the pfsense box even though the box itself is up. I'd try increasing the maximum number of states available provided you have the RAM for it (each state needs about 256 bytes IIRC) or change the state optimization mode to aggressive.

deej1

Hi,

Thanks for your reply, I have actually done both of those things already. it did help and the states are being cleared quick enough (plenty of space for new ones) but it still drops it just takes longer for the firewall to drop.

It can't handle the amount of connections going through it, I'm not sure what else I can tweak to fix this, we don't have problems through other firewalls?

Thanks for your help

cmb

Some of the largest PCI ASV scanners in the world use pfSense at very large scanning scale, it's not a general problem. It definitely sounds like state exhaustion, which can happen in multiple places. Sounds like it's not on your firewall if you definitely have your states high enough, but could be something upstream from it like your modem.

echoranger

The other thing you could do is bypass state tracking entirely for your Nessus device, although you will need to pretty much allow everything to/from that device in pfSense (not sure how applicable this would be to your security policy).

You could create 2 floating rules, one allowing everything FROM your Nessus device and another allowing everything TO your Nessus device. Set them to quick and in the advanced options set state tracking to none. This will effectively bypass the state table for all traffic to/from the Nessus scanner, but again be aware of the security implications of this as pfsense would filter absolutely no traffic for your Nessus scanner.

deej1

Hi,

Thanks for your replies, any idea where in advanced you can disable the state tracking, Ideally I would rather not create an any any rule, but for a quick test it might be ok. but even so will this disable state tracking?

When I login back in to the pfsense from the mgmnt port after the network has crashed the states are about 100/1900000 so I don't think it is to do with the states, I doubled the memory to 4gb and added a second processor the hardware isn't the issue either.

I am running out of ideas here.. :(

Thanks for your help

deej1

I just set state type to none in the firewall but after that no connection could be made to the internet so I had to put it back to keep state.

Any other suggestions??

Thanks

echoranger

I can't tell 100% from your replies but it sounds like you created a single any-any rule with state tracking disabled. If this is the case, this is not what you want to do.

You'll want to create 2 floating rules. One with a source of your Nessus scanner IP and destination of any, TCP/UDP protocols, enable quick, state tracking set to none. Second rule would be a source of any and destination of your Nessus scanner IP, same setup as the above rule.

Sorry if this is actually what you did, from your post its difficult to tell. You might want to post a screenshot of the rules you are creating so they can be looked at.

cmb

It's impossible to do no state in most circumstances like that (anything with NAT), don't do it.

wallabybob

@deej1:

Any other suggestions??

I haven't seen a follow up to: @deej1:

- I will look into these logs more and post again, the logs on the pfsense webgui had nothing in them.

Note though that you need to look at the logs BEFORE rebooting pfSense because some versions of pfSense clear the logs on reboot.

Please also post the output of pfSense shell commands:```

/etc/rc.banner

ifconfig -a

netstat -m


What version of pfSense are you using (from report on the home page)?

You have reported that when the "fall over" condition occurs you can access the pfSense console, on the pfSense console get ping responses from internet hosts, have plenty of free states, client computers can't access the internet. This suggests to me a possible connectivity problem between client computers and pfSense. That this connectivity problem is apparently rectified by rebooting pfSense suggests it is probably in pfSense rather than the client(s?) or something between client and pfSense.

What is between the clients and pfSense?

Does the problem affect all clients?

When pfSense has "fallen over"
1\. What is reported on the client when you attempt to ping an internet host by IP address (say 8.8.8.8, a Google name server)?
2\. What is reported on the pfSense console when you attempt to ping a client computer?

deej1

Hi,

Sorry the logs are always empty, they don't have anything useful in them at all.

Version:
2.0.1-RELEASE (i386)
built on Mon Dec 12 17:53:52 EST 2011
FreeBSD 8.1-RELEASE-p6

$ ifconfig -a

bge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:16:35:9f:db:10

	inet 10.66.3.1 netmask 0xffffff00 broadcast 10.66.3.255

	inet6 fe80::216:35ff:fe9f:db10%bge0 prefixlen 64 scopeid 0x1 

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)

	status: active

bge1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:16:35:9f:db:11

	inet6 fe80::216:35ff:fe9f:db11%bge1 prefixlen 64 scopeid 0x2 

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)

	status: active

fxp0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=9 <rxcsum,vlan_mtu>ether 00:08:02:cd:57:94

	inet 10.66.0.3 netmask 0xfffffff8 broadcast 10.66.0.7

	inet6 fe80::208:2ff:fecd:5794%fxp0 prefixlen 64 scopeid 0x3 

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)

	status: active

fxp1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=9 <rxcsum,vlan_mtu>ether 00:08:02:cd:57:95

	inet 10.66.0.9 netmask 0xfffffff8 broadcast 10.66.0.15

	inet6 fe80::208:2ff:fecd:5795%fxp1 prefixlen 64 scopeid 0x4 

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)

	status: active

lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384

	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 

	inet6 ::1 prefixlen 128 

	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 

	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460

	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1

pflog0: flags=100 <promisc>metric 0 mtu 33200

enc0: flags=0<> metric 0 mtu 1536

bge1_vlan1101: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=3 <rxcsum,txcsum>ether 00:16:35:9f:db:11

	inet6 fe80::216:35ff:fe9f:db10%bge1_vlan1101 prefixlen 64 scopeid 0x9 

	inet 10.66.1.1 netmask 0xffffff00 broadcast 10.66.1.255

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)

	status: active

	vlan: 1101 parent interface: bge1

bge1_vlan1105: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

	options=3 <rxcsum,txcsum>ether 00:16:35:9f:db:11

	inet6 fe80::216:35ff:fe9f:db10%bge1_vlan1105 prefixlen 64 scopeid 0xa 

	inet 10.66.5.1 netmask 0xffffffc0 broadcast 10.66.5.63

	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)

	status: active

	vlan: 1105 parent interface: bge1</full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast>

$ netstat -m

1436/1894/3330 mbufs in use (current/cache/total)

1435/1595/3030/25600 mbuf clusters in use (current/cache/total/max)

1434/998 mbuf+clusters out of packet secondary zone in use (current/cache)

0/90/90/12800 4k (page size) jumbo clusters in use (current/cache/total/max)

0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)

0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)

3230K/4023K/7253K bytes allocated to network (current/cache/total)

0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

0/0/0 requests for jumbo clusters denied (4k/9k/16k)

0/8/6656 sfbufs in use (current/peak/max)

0 requests for sfbufs denied

0 requests for sfbufs delayed

0 requests for I/O initiated by sendfile

0 calls to protocol drain routines

There is switch between the pfsense and users.

The problem affects all clients.

The client computers lose there IP and any network connectivity.

The pfsense box cannot ping 8.8.8.8,

Hope that is enough info, Thanks for your help

wallabybob

@deej1:

Sorry the logs are always empty, they don't have anything useful in them at all.

I find this so surprising I need clarification: When you write "the logs are empty" do you mean "the logs contain nothing at all" or do you mean "the logs don't report anything that seems relevant to this particular problem"?

@deej1:

Hope that is enough info, Thanks for your help

Thanks for the additional information. Unfortunately it is not enough for me to be able to identify the problem.

When I asked @wallabybob:

When pfSense has "fallen over"
1. What is reported on the client when you attempt to ping an internet host by IP address (say 8.8.8.8, a Google name server)?
2. What is reported on the pfSense console when you attempt to ping a client computer?

I was looking for more details than @deej1:

The pfsense box cannot ping 8.8.8.8,

Ping can report a number of different errors and the exact text of the report contains considerably more information than the high level summary "cannot ping". Please provide the details I asked for.

Your report @deej1:

The pfsense box cannot ping 8.8.8.8,

seems to contradict your earlier report that you can ping from pfsense shell to an external website. Maybe the details of the ping response will explain the apparent contradiction. Can you explain this apparent contradiction? (Note I get ping response from 8.8.8.8 over the public internet.)

None of your pfSense interfaces has a public IP address. So what is between pfSense and the targets of the nessus scan? What is between pfSense and the public internet?