Issue with pfsense, internet disconnect

wallabybob

It sounds as if "pfSense froze" MIGHT be better described as "pfSense lost connection to the internet". Until proven otherwise, lets assume that is the case. Then pfSense can still be accessed from the LAN side. With a small investment in training, a person can probably be taught how to capture the pfSense logs from a computer on the LAN side, then reboot pfSense. Alternatively, you could configure pfSense to log to a syslog server running on another computer in your network.

@wallabybob:

Alternatively, you could configure pfSense to log to a syslog server running on another computer in your network.

this sounds like a better plan, the user on site is not always the same user. the users that help me have a hard time finding the power button on the front of the case (which in this case is pretty much the only button on the front of the case). i dont think i will be training them to capture anything.

stephenw10

Haha. :D
Time to get some new minions!

http://doc.pfsense.org/index.php/Copying_Logs_to_a_Remote_Host_with_Syslog

Steve

i configured it to send to a syslog server, last night after my last post.

here is a pic

stephenw10

Hmm, not much helpful there. At what time did it loose connection?
The rrdtool error is normal.

Steve

@stephenw10:

Hmm, not much helpful there. At what time did it loose connection?
The rrdtool error is normal.

Steve

it lost connection on 2-14-13 at 322am. i rebooted the server and then enabled syslog so you wont see it in there since the log wiped it self or only saves 50 records? not sure, i just started looking at the log.

but if it locks up again (from the outside in) hopefully it would have sent something to the syslog server and i can update the thread. it was weird that it happened two times in the last few weeks and never happened prior to that.

thanks.

stan-qaz

There is another topic around here somewhere on the WAN interface getting a non-routable IP from a rebooting modem that you might want to look for.

I was looking for an easy fix, better than rebooting pfSense or the modem and I think going to the https://pfsense.home/status_interfaces.php page and clicking the Release/Renew button got her back on line.

@stan-qaz:

There is another topic around here somewhere on the WAN interface getting a non-routable IP from a rebooting modem that you might want to look for.

I was looking for an easy fix, better than rebooting pfSense or the modem and I think going to the https://pfsense.home/status_interfaces.php page and clicking the Release/Renew button got her back on line.

how do i do that when i am not on site?

also, i checked the logs of the ddns client running, IP has been the same for the last 10 months.

stan-qaz

If you aren't getting a non-routable ip in your pfSense logs then this may not be the issue.

http://forum.pfsense.org/index.php/topic,57353.msg306354.html#msg306354

Another modem related possibility here:

http://forum.pfsense.org/index.php/topic,57258.0.html

If you think your users couldn't click the button or you don't trust them with the password then one of the cron based suggestions from the second link (page 2) might do for you.

Trying a snapshot version to get this fix is also a possibility.

https://redmine.pfsense.org/issues/2792

tim.mcmanus

I had log entires like the ones you listed. Try grabbing the latest 2.0.3 update. I'm running the Feb. 9 update and it's been rock solid. Did you ssh into the box and clog /var/log/system.log?

@tim.mcmanus:

I had log entires like the ones you listed. Try grabbing the latest 2.0.3 update. I'm running the Feb. 9 update and it's been rock solid. Did you ssh into the box and clog /var/log/system.log?

nope, not sure how to do that.

Downloading new version information…done
Obtaining current version information...done

You are on the latest version.

2.0.2-RELEASE (i386)
built on Fri Dec 7 16:30:38 EST 2012
FreeBSD 8.1-RELEASE-p13

You are on the latest version.

stephenw10

2.0.3 is only available as a pre-release snapshot currently. It is ready for release though, just waiting for an upstream SSL patch.
See: http://forum.pfsense.org/index.php/topic,58203.0.html

Steve

@stephenw10:

2.0.3 is only available as a pre-release snapshot currently. It is ready for release though, just waiting for an upstream SSL patch.
See: http://forum.pfsense.org/index.php/topic,58203.0.html

Steve

i ended up going to 2.1 beta

2.1-BETA1 (i386)
built on Fri Feb 15 15:43:55 EST 2013
FreeBSD 8.3-RELEASE-p5

You are on the latest version.

tim.mcmanus

@tomdlgns:

@tim.mcmanus:

I had log entires like the ones you listed. Try grabbing the latest 2.0.3 update. I'm running the Feb. 9 update and it's been rock solid. Did you ssh into the box and clog /var/log/system.log?

nope, not sure how to do that.

The webUI on pfSense is really good, but like anything else it's limited in what it can do. In order to get more, or arguably better, information from pfSense you need to log into it and run some command line stuff. There are a couple of ways you can do this:

In you are on a Mac or Linux desktop, you can enter into the command line program (Terminal on the Mac) and then type the command ssh psfsenseadministratorlogin@pfsenseIPaddress. Obviously you want to substitute your pfSense administrator login and your installation's IP address. Something like ssh admin@192.168.1.1. Once you've logged into the box you'll see the same screen as you would if you had a monitor connected to your pfSense box. If you cannot ssh into your pfSense box but have a display connected to it, these instructions will apply at this point. Select option #8 and enter into the shell.

Once you're in the shell you'll see a different command prompt indicating that you're in a shell. At this point you can enter in the command clog /var/log/system.log. This will print out the contents of the system.log file on the screen. It'll blast across your screen, and if you're logged in via a Terminal program you'll be able to scroll up to see the entire history. Here is the pfSense Wiki page about it (Wiki Link).

If you want to look at some of the other logs on the system, you can issue the shell command ls /var/log/ to see a listing of the log files in the /var/log directory. To view them, use the same clog /var/log/nameoflogfile.log. I recommend getting an ssh program (there are many out there for Windows) if you don't already have one. It will make it easier to cut and paste some of this log data and to view lengthy log histories.

Now, updating to a more current version requires a little bit of searching but it's a handy thing to know. The auto update feature in pfSense that's configured out of the box only grabs the latest "official" release. So if you are currently running 2.0.2, all it will see is the 2.0.2 release since that's the latest "official" release. There are different releases available, and right now let's focus on getting the 2.0.3 release.

You will be downloading and installing a Snapshot of a more current release. These are tested version but not yet official releases. It can be risky to download these releases if they are very early on in their development, but right now the 2.0.3 development is nearly done and it is very stable. So there's less risk downloading this snapshot at this time, and there's a very good change it will remediate the issues you're seeing.

The URLs for the snapshot servers can be found at the beginning of this thread (thread link). Choose either the i386 thread or AMD64 depending on your build. It looks like you're using the i386 build, so go to that link. On the snapshot server page, select the third link to grab an update. This will bring you to a directory listing. You'll notice that all of the files have the word PRERELEASE in their names, this is okay, it just lets you know that it's not an official release. I grabbed the Feb. 9 release, you could grab a more current one if you'd like. In my case I scrolled to the bottom and grabbed the pfSense-Full-Update-2.0.3-PRERELEASE-amd64-20130209-2111.tgz file. That's what I used to build my box. Find the update that best aligns with your installation and download it. Be sure to grab the large .tgz file and do not try to decompress it.

Once you have the right file you can go to the pfSense webUI. Go to System->Firmware and enable Manual Firmware Upgrade by clicking the button. Then click on the Choose File button and select the PRERELEASE…tgz you just downloaded. Click on the Upgrade firmware button. You'll get a dialog, read it and click okay or continue to start the upgrade process. I probably should have said this at the beginning of the post, but BACKUP YOUR SETTINGS!!! You never know what might happen and you may need to reinstall the server from scratch. This is extreme but it's less painful if you've backed up your setting in advance. I've killed my pfSense box enough times to have this lesson burned into my skull.

The upgrading process should go smoothly and the box will reboot itself. After a minute or so log back into the webUI and check out the system logs (or ssh into the box). See if your crashes persist. If they do, go back to the system logs and see what they say. Post the results here, and I'm sure someone will be able to get you a little further along in the process.

Best of luck!

@tim.mcmanus:

@tomdlgns:

@tim.mcmanus:

I had log entires like the ones you listed. Try grabbing the latest 2.0.3 update. I'm running the Feb. 9 update and it's been rock solid. Did you ssh into the box and clog /var/log/system.log?

nope, not sure how to do that.

Best of luck!

wow, great post, thanks.

i am on the beta 2.1, do you guys recommend getting off that and getting the 2.0.3 release?

everytime i make a change in pfsense i save the config and then i save the config and i label it. i learned by having to reformat the box when i was first starting and didnt have a recent backup, now i have many.
i have a mac, but i also have a windows laptop that runs putty, i ssh/telnet with that program on other devices.
i also have a monitor connected to pfsense, but putty is still nice, wouldnt require me to be at the pfsense terminal and i can copy/paste/save the logs on my machine and read through them. thanks for the extra info and wiki links.

tim.mcmanus

You should be okay with the 2.1 beta, but it is a beta, so keep that in mind.

I chose the 2.0.3 update to minimize any risk I may be inflicting on my network. However, there are people using 2.1 in production with excellent results. I am too risk averse and stayed conservative with the 2.0.3 release.