Overspec Firewall Keeps Dropping internet connection.

whatthehost · Jun 19, 2012, 12:16 AM

Hi There

I recently upgraded our firewall here, from an old Linux server running shorewall through webmin, to a new server running pfSense, when the hard drive died. I have been having ongoing problems ever since, with the firewall randomly dropping the internet connection every two or three days.

The new server is a Quad Core Xeon on an intel board with 4GB of Ram, running two NICs, using NAT for external access to our websites and mail server.

We are on an 6MB/6MB Telstra Fibre connection and we had no issues with it on the previous setup.

I disabled gateway monitoring and removed what appeared to be a duplicate gateway, now there is only one, to no avail.

Can anyone shed any light on this, or tell me where to start looking. As a Linux guy, I am not totally up with the play on BSD.

Thanks
Sam

wallabybob · Jun 19, 2012, 4:00 AM

@whatthehost:

We are on an 6MB/6MB Telstra Fibre connection and we had no issues with it on the previous setup.

"Previous setup" meaning Shorewall OR pfSense with the now dead hard drive?

@whatthehost:

Can anyone shed any light on this, or tell me where to start looking. As a Linux guy, I am not totally up with the play on BSD.

A good start would be to look through the pfSense system log around the time the pfSense WAN link went down. See Status -> System Logs for the most recent system log entries or give the pfSense shell command```

clog /var/log/system.log


Does your WAN link use PPP or DHCP or …?

What version of pfSense have you installed? There have been reports of PPP not restarting on some WAN links but it has proved difficult to reproduce the problem. (For example, I haven't seen it in the more than a year I have used PPP on my WAN link.)

whatthehost · Jun 19, 2012, 7:07 AM

Hi Thanks for the reply.

@wallabybob:

@whatthehost:

We are on an 6MB/6MB Telstra Fibre connection and we had no issues with it on the previous setup.

"Previous setup" meaning Shorewall OR pfSense with the now dead hard drive?

Shorewall

@whatthehost:

Can anyone shed any light on this, or tell me where to start looking. As a Linux guy, I am not totally up with the play on BSD.

A good start would be to look through the pfSense system log around the time the pfSense WAN link went down. See Status -> System Logs for the most recent system log entries or give the pfSense shell command```

clog /var/log/system.log
Yeah didn't find anything telling in the logs.

Does your WAN link use PPP or DHCP or …?

It's a static business ethernet connection, over fibre.

What version of pfSense have you installed? There have been reports of PPP not restarting on some WAN links but it has proved difficult to reproduce the problem. (For example, I haven't seen it in the more than a year I have used PPP on my WAN link.)

I am on 2.0.1-RELEASE-pfSense (amd64)

I haven't had much time to see what's going on, when it happens, as I have to get the net back up ASAP. But it appears to be lagging on the web interface, however it shows miniscule CPU and Memory usage.

Thanks
Sam

wallabybob · Jun 19, 2012, 11:31 AM

On the Interfaces -> (assign) page what network port is reported for the LAN interface? the WAN interface?

Please provide more details of what you mean by the "firewall dropping the internet connection" - browser on LAN clients reports "Host unavailable"? browser downloads stall? browser reports timeout?

What do you do to recover from "the firewall dropping the internet connection"?

The next time it happens I suggest you issue the following commands in a SSH session to the pfSense box (preferred) or on the pfSense console (if you can't get a SSH session to the pfSense box)```

ping -c 3 www.bigpond.com

ping -c 3 144.135.18.32


As yet there isn't enough evidence to distinguish between

*   loss of communication between pfSense WAN interface and the Internet

*   loss of communication between pfSense LAN interface and local systems

*   name server inaccessible

*   etc

whatthehost · Jun 19, 2012, 11:15 PM

WAN is em0
LAN is em1

When it drops, I can ping the firewall. I can log in to the firewall, although the web interface is laggy. I can't ping external IP's or DNS names from inside the network or from the firewall itself.

I have to restart the firewall to recover.

I have tried pinging when it goes down and the results were as above. Can't ping addresses or names, so not a DNS issue.

Thanks
Sam

cmb · Jun 19, 2012, 11:47 PM

What type of connection? DHCP, PPPoE, static, …?

whatthehost · Jun 20, 2012, 12:03 AM

Static ethernet on a fibre connection. This morning, new problem, the internet was still working. NAT to our Web server stopped working. NAT to other servers worked as usual. Could telnet server locally on web ports, not from outside. Restarted firewall, started working again. Maybe it doesn't like our hardware. Either way pfsense is coming out tonight. Can't keep this up, the boss and users are losing patience.

cmb · Jun 20, 2012, 12:07 AM

Sounds a lot like an IP conflict, like your former firewall still plugged in or something else with those IPs assigned. Rebooting "fixes" because it sends a gratuitous ARP which makes that system win back the conflicting IP at least for some period of time. Lag in some parts of the web interface happens when you have no DNS connectivity, 2.0.2 and newer won't though.

whatthehost · Jun 20, 2012, 12:18 AM

I don't think it's an IP conflict, as the former firewalls motherboard blew up and was replaced with pfSense. Only way it could be a conflict is if someone at Telstra was encroaching on our subnet. Wouldn't pfSense log detected conflicts though?

I will install it on a virtual machine on one of our vmware servers. That will prove/disprove hardware conflict.

cmb · Jun 20, 2012, 12:23 AM

You will see that, when it happens, in the system log for the WAN IP itself and for IP aliases and CARP IPs, that is assuming the firewall can actually see the conflict. Depending on the type of connection and other details of the network it can be possible for an IP conflict to happen and only be seen by your upstream router, though that's usually not the case.

whatthehost · Jun 21, 2012, 3:13 AM

I was using IP Alias not CARP. Should I be using CARP or PROXY ARP instead?

It silently dropped inbound web traffic to one host this morning and fixed itself after about 15 minutes. No errors in log. Seems consistent with ARP problems.

Thanks
Sam

cmb · Jun 21, 2012, 5:30 AM

CARP is fine too. CARP will log IP conflicts as well. Proxy ARP won't. What it can't tell is where there is an MAC conflict on the network. CARP, like VRRP, uses a virtual MAC address that's determined by the VHID. If anything on the same broadcast domain is using the same VHIDs as you are with either CARP or VRRP, you have a MAC address conflict, which will exhibit itself the same as an IP conflict. If you're using low numbered VHIDs, I'd change those to higher uncommonly used VHIDs, 200+ generally a good choice in such circumstances.

whatthehost · Jun 28, 2012, 12:41 AM

Ok thanks

Virtualising the firewall, has (so far) stopped it from intermittently dropping all traffic. I have had one more instance of it stopping forwarding under proxy ARP. Have shifted it back to IP Alias and will see hot it goes.

Cheers
Sam